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ABSTRACT 

The prime objective of this review was to examine the 
existing literature relating to proposal development and evaluation 
in order to establish a perspective on any empirical base underlying 
the process. Findings related to seven areas are presented: preparing 
the proposal, utilization of support services, preparation cost and 
return relationships, reviewing and evaluating proposals, 
establishing credibility of the peer review process, proposal quality 
and program success, and perceptions and attitudes. Observations on 
both substance and methodology are synthesized, and 
few empirical studies have been directed toward the 
proposal preparation; (2) support services provided 
developers found to be most useful focus upon the somewhat mechanical 
aspects of a proposal; (3) training in proposal development is a 
justifiable service and cost; (4) the return on investment justifies 
the costs of development; (5) the decision points in the development 
process within an institution should be the object of careful study; 
and (6) the predominant method for research on proposal development 
tended to be some form of correlational analysis. (BW) 
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PROPOSAL DEVELOPMENT AND EVALUATION: 
A SYNTHESIS OF EMPIRICAL STUDIES 



I. INTRODUCTION 

The past several decades have witnessed both inside and 
outside the field of education an increase in activity 
relating to the acquisition of external resources to support 
internal research, development, and service efforts. The 
roots of this resource seeking activity lie in the adoption 
by many federal and other funding agencies of the "project- 
concept. To secure external funding, the dimensions of a 
proposed project have to be outlined in a document referred 
to as a "proposal." As the amount of funds available has 
increased over time so has the activity of developing 
proposals to secure access to these funds. Thus, a pattern 
of professional activity has developed on both an individual 
and institutional basis devoted to the task of writing 
proposals . 

Given that this process of proposal creation has become 
very extensive, it would seem logical that the task of 
proposal development should be based upon principles and 
practices derived in so far as possible from empirical 
studies. What principles do exist appear to be based upon a 
"show and tell" or "folklore" approach wherein an 
individual's views are expressed on the best procedure to 
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make sure one -gets funded.- Such self-reports may be of 
value especially if one were able to accumulate the 
individual experiences into some form of collective wisdom. 
Even if that were brought about , the establishment of 
principles and practices based upon valid and reliable 
empirical evidence would seem to provide a more desirable 
base of knowledge. 

The purpose of this study was an initial attempt to 
determine the nature of empirical studies relating to 
proposal development existing in published and retrieval 
form. As such, the study was considered as an exploratory 
"state-of-the-knovledge w investigation. The results of such 
a study would be helpful not only in establishing a 
knowledge base but would also provide some perspective on 
what issues or problems have been studied, how they have 
been approached, and what studies might be initiated in the 
future. This report presents the results of that 
exploratory investigation . 

The general literature relating to proposal development 
is quite extensive and covers many substantive and 
discipline areas from human services to military 

procurement. Within that literature base, the focus here 
was to be upon two major aspects of it* First, a focus on 
what were chosen to be called -empirical studies.- Such 
studies were those wherein an investigation was undertaken 
to answer a question of interest or to test a hypothesis, 
where data were collected and analyzed, and conclusions 
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drawn. Such studies could take the form of self-report 
surveys, interviews, £x. ££££ facto analysis, or experimental 
investigations. Literature reporting on personal 

observations alone were not to be included. Selected 
reports from several federal agencies reporting on proposal 
evaluation were included because the information presented 
was deemed to be of sufficient interest to those developing 
proposals. Second, there was to be a major focus upon the 
field of education in view of the major utilization to be 
made of the results. Studies and reports from areas outside 
of education were included if they fell within the general 
definition of an empirical study, the content was 
correlative with the processes involved, and were consistent 
with the purpose of the study. 
Data Collection 

The basic process of data collection consisted of 
identifying studies and reports, securing original documents 
where possible, reviewing their contents for suitability, 
preparing notes and abstracts, assembling them into similar 
or related topics, and then fitting them into a framework 
for presentation and synthesis. 

The process of study identification was done 
principally thru the use of information storage and 
retrieval systems and searching data bases that would most 
likely include reports dealing with proposal development. A 
profile of basic and related terms (see Appendix A) was 
developed with the assistance of the Mechanized Information 
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Center of The Ohio State University. Using this profile, 
the data bases searched were Dissertation Abstracts, 
Educational Resources Information Center (ERIC) , Management 
Abstracts, National Technical Information Service (NTIS) , 
Psychological Abstracts, and Sociological Abstracts. No time 
frame limited the search of the data bases. In addition to 
searching mechanized data bases, letters were sent to 
selected personnel located principally in the field of 
education requesting them to forward the titles of any 
studies with which they were familiar. 

The process of study identification had a form of 
"ripple effect." One reference would make note of a related 
stud'y. In such cases, efforts were initiated to secure a 
copy of the related document. Whenever possible primary 
sources were reviewed instead of relying on abstracts or 
secondary documents. 

Each document identified and secured was examined first 
to determine its relevancy to the study. If accepted, it 
was screened for empirical data organization and analysis. 
Suitable studies were then categorized as to the process 
step in proposal development and evaluation to which it 
mainly referred. Bibliographical information was developed 
along with abstracts and notes to be used in generating the 
basic report. All summaries and notes used in the analysis 
were prepared solely by the author. 

The data collection period approximated the time from 
September 1982 to August 1983. Even with the use of 
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agencies such as interl ibrary loan and related services, 
delays were encountered in securing some documents and some 
identified reports were just not available thru normal 
change channels. Personal letters addressed to the authors 
were often returned as not deliverable. 
Organizi ng Structure 

As noted above, each study was placed in a structure 
reflecting the general process steps in proposal development 
and evaluation. proposal development was the general 
category for studies and reports relating primarily to the 
task of creating a proposal within an institutional setting. 
Proposal evaluation was the category involving studies and 
reports relating to the review, evaluation, and funding of 
proposals after submission to a support or funding agency. 
Subtopics within each of these two principal areas were 
developed to highlight the particular emphasis of one or 
more studies. A section on attitudes and beliefs about the 
process as a totality or in part was included as a result of 
identifying studies seeking such information. After the 
major task of categorizing and summarizing the reports was 
completed, a synthesis of major findings was established to 
reflect the "sta te-of -the-knovledge " . The synthesis was 
used to support a series of conclusions and implications 
regarding the process of proposal development and 

eval uation . 
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II. PREPARING THE PROPOSAL 
The prime task or work effort focuses upon development 
of the actual proposal. There are any number of books, 
brochures, pamphlets, and guidelines derived from past 
experience, both individually and collectively, to provide 
the prospective proposal developer with assistance. The 
available literature is rather voluminous and cannot be 
reviewed here. In searching this same literature for 
studies and reports relating to actual proposal preparation, 
the process appears not to have been highly investigated. 
Six studies were identified that could be considered as 
being related to the actual tasks of proposal preparation. 
Three of the studies, however, are probably best classified 
as case study reports describing the relationship between 
proposal development and other institutional considerations, 
ini tiating Proposal D evelopment 

In the typical case, proposals are created as a 
response to either some form of general program announcement 
or to a Request for Proposal (RFP) issued by a funding 
agency. The RFP case often restricts the proposal writer in 
terms of the dimension of how a response should be 
addressed. Further, the response would be controlled to 
some by degree by the need to understand what the RFP is 
requesting. 

The ability to comprehend an RFP was investigated by 
Milbrandt (1979) in a study concerned with the language used 
in Department of Defense Contracts to determine if such 
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usage hindered communications and understanding. The study 
focused upon determining the readability levels of various 
documents, including RFPs. Using different readability 
fomulas plus personal interviews, the findings showed that 
the RFP had a readability level at the college level by one 
formula, a Dunning Fog Index of 17.7, and a grade level of 
11.7 for the AF formula. Based upon these results plus the 
comments secured from the interviews, particularly of small 
vendors, the author suggested that readability was not a 
problem for the potential vendors. The interviews indicated 
that if there was a problem, resolution was secured through 
the use of professional, technical, and legal staff 
personnel. The author did note that in some cases small 
vendors might not become involved in seeking contracts due 
to the volume of documentation to be understood. The study 
is limited in its reporting since no data is presented as to 
the number of documents of each type reviewed or how many 
actual interviews were conducted. 

As noted, the response to an RFP is controlled by the 
funding agency as to information to be supplied by a 
prospective vendor. Dycus (1976) investigated the impact 
upon proposal evaluation if a developer should present a 
counter argument to the contents of an RFP. Conceptually, 
Dycus operated from the research of Hovland and others with 
regard to the effects of one-sided (supportive defensive 
only communication) against a two-sided (combined supportive 
and refutation defensive communication). The latter 
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approach is considered as being superior. The study 
involved the construction of two mock proposals in response 
to a short statement of work from the Federal qovernment. 
For each proposal, the management plans only were modified. 
One proposal presented only information in response to the 
RFP while the second presented not only the required 
information but presented a counter argument with regard to 
the location where the program tests should be conducted. 
The governments option was refuted and the company's main 
facility was advocated. In both proposals, paragraph and 
work length were equal. 

Using mail procedures, the proposals were rated by 27 
experienced proposal evaluators based on a set of nine 
evaluation criteria using a scale of -3 to +3. One 
criterion was presumed to reflect the response contained in 
the management plans. Order effects were controlled. The 
results showed that the two-sided communication rated higher 
with a mean of 1.5 versus 1.2 but the difference was not 
statistically significant. Dycus concluded that the 
sideness of a communication did not strongly dominate over 
the substance of the message. 

In the article, Dycus references a second study he 
conducted which appeared under the title "The Relative 
Effectiveness of Two Proposal Approaches - An Experimental 
Study" which appeared in Technical Communicat iona in 1975. 
Letters addressed to the author were returned and attempts 
to locate the study through information retrieval systems 
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have not yielded the report. It is not possible, therefore, 
to knew if the same study above was being reported upon or 
if a different experiment had been conducted. 

As part of a project to review and evaluate the Small 
Project Grants Program of the Bureau of Research of the 
Office of Education (DHEV7) , Rogers, Sanders, and Levenson 
(1969) sought information from applicants as to how they 
heard about the program. Since the study is referenced in 
subsequent sections of this report, selected details about 
the study are presented now. 

Briefly, the Small Grants Program operated out of the 
nine Regional Offices of Education with the general purpose 
of stimulating educational research, particularly in small 
colleges and among young researchers. Grants were limited 
to $10,000. Information was sought in 1969 from funded and 
non-funded applicants of the program for fiscal year 1968. 
For the period involved, there were 874 applications with 
298 funded and 585 not funded. Usable completed 

questionnaires were returned from 6 65 appi icants for a 
return rate of 78 percent. In March 1970, questionnaires 
were sent to the 512 field readers who reviewed the 
proposals with 423 responses returned. Details regarding 
follow-up procedures are presented in the report text. Of 
the total report, three sections are of principal interest 
to the current investigation; namely, the proposal contents, 
its development, and its processing after submission. Since 
the section on proposal content details an analysis of such 

13 
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topics as research design, sample size, use of computers, 
statistical analysis procedures, and budgets and not £££ 
on its development, findings relevant to those issues are 
not presented in this report • Findings with regard to 
proposal development and evaluation are presented in this 
report as the results seem relevant to the major section 
invc ' ved. 

Rogers and others solicited responses from applicants 
as to how they heard about the Regional Research Program 
(RRP) , at what point their research plans were formulated, 
and the nature and number of the resources used in preparing 
the proposal. Information about the program was most often 
secured from a colleague or superior (64 percent) followed 
by personal contact with USOE official (12 percent) , oral 
presentation by OSOE official or USOE written materials (10 
percent) , and other sources with lesser percentages. In 
terms of resources available for preparing the proposal, 87 
percent indicated access to the Small Grant guidelines, 72 
percent to a resource person knowledgeable about applying 
for grants, with 45 to 33 percent having available resources 
such as copies of previously submitted proposals, sample 
application forms, information banks of agencies funding 
research, and ERIC materials. Only 10 percent indicated 
they had access to an USOE paper written on how to win a 
grant. Data presented with regard to the number of sources 
used in developing the proposal indicated the proportion of 
applicants funded tended to increase as more resources were 



Page 11 



available running from 36 percent for one source up to 45 
and 46 percent for four or five sources. In contrast, other 
data revealed that having a resource available was more 
highly related to funding than actually using it in 
preparing the proposal. When asked if they had talked with 
an USOE official, only two out of five did so. Of those who 
did, a larger proportion (43 percent) were funded. Of the 
group who talked with the agency official, 96 percent of 
those funded indicated that it was helpful while only 64 
percent of those not funded said it was helpful. The person 
most often met with was the Director of Research (68 
percent) at the several regional offices. When asked if the 
proposa 1 was written on their own time or on working time, 
347 or 52 percent indicated they wrote on their own time. 
Of the group that wrote on their own time, about 1 out of 3 
(36 percent) were funded. Information was sought from 
applicants as to the juxtaposition of their research plans 
to the research program announcement. Of the applicants 
eventually funded, 40 percent indicated that they had a well 
defined plan before thinking of applying; 30 percent only 
had a general idea, and 24 percent developed plans after 
giving thought to applying. Of the total group of 665 
applicants, 544 or 82 percent had well developed plans. 
Even though well developed plans existed, 83 percent had not 
submitted the proposal to another agency prior to submission 
to the £££. 
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Preparation and i nstitutional Relat i onsh i ps 

Two studies were identified that related to proposal 
preparation but in which the focus was upon using the task 
as a means of accomplishing other ends, particularly 
institutional change. Cook (1971) and Anderson (1974) 
describe the process of proposal development as a vehicle 
for bringing about change in local school district 
curriculum improvement. Since both studies are similar in 
nature and focus, the Cook study will be highlighted. The 
decision to seek funds under the Elementary and Secondary 
Education Act, Title III, to meet stated educational needs 
caused a local school district from Superintendent on down 
to become involved in developing a proposal. In his report, 
Cook stressed the conflict between the Superintendent and 
his traditional administrative instrumentalities and the 
changes created by the preparation of the proposal. He 
noted that the proposal brought this condition out into the 
open and would have created morale problems had the 
superintendency not understood the consequence of not 
realizing the time and effort that had gone into the 
proposal development. In her report, Anderson noted that 
the success of the proposal development effort was judged by 
the approvals secured from the several offices involved in 
the development of the proposal, a copy of which is included 
in the Anderson report. 

Doth of these studies are case studies of where 
proposal development rather than being a one-person effort 
ERXC is an in* titutional effort impacting on various 

1fi 
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organizational entities and their role and functions. Both 
reports outlined the general steps followed but such actions 
were viewed by both authors as being secondary to the 
function that the proposal development activity played in 
modifying the organizational structure. 

In addition to the studies described above, a third 
study related to the preparation process in institutional 
settings should perhaps be included in this section. As 
part of the Research Management Improvement Program of the 
National Science Foundation, Jebens, Millstein, and Wearly 
(1975) and Jebens (1976) report on a study of the decision 
points and responsibility areas associated with the 
institutional internal review of proposals developed and 
submitted using both UCLA and the University of 

California-San Diego as data sources. Narrative 

descriptions of the process of generation and review of 
proposals along with flow chart diagrams of the decision 
process are included. Most comments are of a narrative 
nature but some empirical data is presented with regard to 
the budgetary dimensions of proposals. Based upon an 
analysis of 100 percent of contracts with all federal 
agencies and a 33 to 50 percent sample of NIH and NSF .grants 
for 1974, median changes between requested and awarded funds 
for 60 contracts from seven different agencies centered 
around 21 percent; that is, about half of the contracts had 
changes greater than 21 percent and half had less. 
Approximately one out of six contracts from seven agencies 
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showed changes in one or more budget categories. As for 
grants , 146 grants from the seven agencies showed changes of 
about 24 percent in total budget and about 40 percent 
changing one or more budget categories. No information was 
presented in the report with regard to any changes upward or 
downward only that the categories and total did change. 
Similar data are presented for change between award 
allocation categories and actual expenditures. It was noted 
that there were significant changes for a large percentage 
of the cases and such changes were to a fair degree large in 
comparing requested to allocated funds. The report contains 
comments relative to a variety of topics relative to 
proposal preparation and its relation to areas such as 
teaching and instruction, research and university mission, 
need for proposal training, the assignment of institutional 
responsibilities for proposal review and clearances 
including the function of the principal investigator. This 
report is recommended reading for persons interested in 
gaining insight into the processes and issues associated 
with proposal preparation within a large university complex. 
Coynmentary 

Considering the large amount of time and resources 
devoted to proposal preparation within organizations, it was 
interesting to observe the lack of studies directed toward 
such issues as to how the task should be approached, formal 
and informal procedures for responding, or the value of 
staff development training sessions. The studies reviewed 
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suggest that individuals can comprehend requests for 
proposals, initiate responses to them, and often have plans 
developed in some form prior even to an RFP or program 
announcement release. The impact upon changes in 
institutional behavior, including the internal review 
decisions and organizational commitments, were observed to 
be present. There appears, however, a lack of studies which 
can offer suggestions as to how an individual and 
organization can best proceed with the proposal preparation 
process. 

III. UTILIZATION OF SUPPORT SERVICES 
As a means of facilitating the process of proposal 
development and submission, it is a common practice for most 
organizations to establish some structure of support 
services. These service offices vary in size from a 
part-time assistant to extensive offices consisting of a 
number of full-time, specialized staff members representing 
various disciplines and services. Both types of offices not 
only assist in seeking grants and contracts but often carry 
out monitoring operations after funding. Three studies were 
located which focused upon the use of and value of such 
support services. One study isolated the costs of providing 
a specific kind of service. 
Frequency and Value of Services Used 

Harty (1977) reported on faculty preferences for 
support services at a large midwestern university. A 33 
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item Likert-type scale was constructed listing 6 major 
service areas and 27 specific service activities. Each 
service was to be rated from Unimportant to Most Important. 
The scale was distributed to a random sample of 70 from a 
total of 228 faculty members. A total of 65 forms were 
returned with 59 being usable for analysis. Demographic 
data regarding the level of proposal writing activity and 
funding, frequency of assistance sought, time at the 
institution, academic rank, major responsibility, cognate 
area, and tenure or non-tenure status was secured. Analysis 
was made for each level of response within each demographic 
characteristic with mean ratings serving as indicators for 
identifying level of importance. The Most Important general 
service activities over the several demographic 

characteristics were those of Proposal Budgeting, Faculty 
Interactions, and Funding Information in that order. The 
Least Important general service activities were Proposal 
Development, Proposal Routing, and other services. Specific 
activity items rated as Most Important were such items as 
Distribution of Seed Money, Funding Information for Specific 
Projects, and Information on Institutional Costs. Items 
rated as being of lesser importance related to Information 
about Postal Deliveries, Access to Style Manuals, and Sample 
Letters of Fund Solicitation. In summarizing the results, 
Barty noted that faculty are not too concerned with the 
mechanical aspects of proposal development, that the faculty 
more experienced in proposal writing have developed time 
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saving procedures, and that the greatest disagreement 
regarding importance of services was related to academic 
ranks and cognate «~reas. 

A similar investigation involving both preparation 
services and post-funding support was conducted by Fiedler 
(1979) at a large western university. Osing an open-ended 
questionnaire approach, a randomly selected set of 192 
faculty members involved with scholarly research was 
interviewed. Most of the respondents were from the science 
fields, were full professors, and 86 percent had applied or 
were applying or planning to apply for external funding. 
The respondents were given a list of services and asked to 
rate their use and the quality of service provided. 
Responses were coded and mean ratings determined for 10 
services with 1 being high and 5 low. In addition to the 
use and quality of each service, respondents were also asked 
to indicate the source from which they usually sought such 
services. The number indicating each of the services varied 
so both frequencies and percents were reported. 

The most frequently mentioned services used were 
Clerical/Secretarial Assistance and Duplicating Services 
with mean ratings of 1.9 for Clerical assistance and 2.0 for 
Duplicating assistance* Ratings of quality of service were 
2.0 for Clerical Assistance and 1.8 for Duplicating 
Services. The item least frequently mentioned and also 
rated lowest in importance with a mean rating of 3.6 was 
that of Editorial Assistance in Presenting Data in 
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Proposals. At the same time, this service had a mean rating 
of 1.8 in terms of quality. Items in between these two 
extremes in order of relative mean ratings were Assistance 
with Reporting, Budget Monitoring, and Accounting; 

Accounting Assistance in setting up budgets and costs 
estimates; Information on Agency Requirements; Information 
on Sources of Funding; Notification of RFP's; Information on 
Available University facilities; and Notification of Current 
status of proposals and their progress through the grant 
process. The services rated highest in quality provided in 
order were Assistance in Setting up Budgets, Duplicating 
Services, and Editorial Assistance in Data Presentation 
being rated equally high with mean ratings of 1.8. Other 
services rated in terms of being lower in quality were 
Clerical Assistance in Writing Proposals, Information Agency 
Requirements; Information on Available University facilities 
and resources; Information on Sources of Funding; 

Notification of RFP's; and Assistance with Reporting, 
Budgeting, Monitoring, and accounting; with the lowest rated 
item in quality being Notification of current status of 
proposals and their progress. Although the responses varied 
with the type of service, the faculty member's own 
department was the most frequently indicated source of 
support. When asked about which units could provide 
services, the highest rated was the library and computer 
centers followed by duplicating, printing, and publications. 
Additional data is presented by the author with regard to 
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perceptions about rewards for seeking of funds, issues 
relating to knowledge about the institutional proposal 
procedures, and items relating to the general areas of 
research administration and management. 

In her discussion of the results, Fiedler noted that 
the respondents believed the university should provide a 
variety of services, including training in proposal writing- 
While there are numerous articles suggesting the nature of 
such training, only one report was found reporting empirical 
data on training. Rainey (1974) conducted a study to 
determine the value of proposal development training. A 
survey form asking about the importance and nature of 
proposal development was sent to a random selection of 50 
manufacturing firms doing a minimum of $1,000,000 in sales. 
The corporate executives were asked to indicate the 
importance of the ability to write both intra- and 
inter-company business proposals, A second survey was sent 
to college teachers of business communications who were 
members of a professional association. They were asked if 
students were given any time on proposal writing in the 
course and how beneficial such training would be. 

Of the business respondents, 50 percent indicated that 
proposal writing was of Great or Crucial Importance while 65 
percent indicated that proposals play an important part in 
the operations of the firm. Replies were received from 116 
university professors. Thirty-eight instructors (33 

percent) thought proposal writing was important enough to 
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have a separate course on the topic. However, only 44 
percent thought such a course would be moderately or very 
beneficial. Some time was given to proposal writing only in 
courses conducted by 21 teachers. 

Using the information from the surveys plus a review of 
the literature on proposal writing along with information 
from informal conversations with both businessmen and 
students , Rainey offers ten untested hypotheses about 
proposal writing and its role in undergraduate business 
training along with behavioral objectives for a proposal 
unit and a list of student activities. 
fost of Information Services 

To facilitate the seeking of sources of funds, most 
institutions arrange to have some support service in the 
form of a "library" or information system identifying 
agencies which accept proposals. Hensley and Williams 
(1981) examined the utility of such a library by relating 
its use to costs, a point not previously noted in prior 
studies dealing only with use. They developed a means of 
recording costs associated with the three main functions of 
the system; Developmental or the providing and disseminating 
of information about funding opportunities; Administrative 
or the obtaining and dissemination of information about 
project management; and Acquisition or soliciting and 
staring of information related to research development and 
administration. Costs wefe obtained and assigned to the 
three respective functions. For the fiscal year 1980, the 
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total direct library costs were $38,683 with about one-third 
of the costs going to professional salaries. If an indirect 
cost rate of 53 percent is added, the total costs were 
$59,224. In terms of function, about 71 percent of direct 
costs was spent on Development, about 11 percent for 
Administration, and about 18 percent for Acquisisiton 
function. The authors also attempted to isolate the costs 
associated with targeting information to selected faculty 
members. They distinguished between general information 
costs and direct targeted costs. Using data from the costs 
of library functions, they noted that the general 
information costs were about $7,700 a year while targeted 
costs were about $14,735 per year which results in a 
per-unit cost of approximately a dollar per bit of 
information. The authors concluded that the costs of about 
$10 per year per faculty member for general information 
distribution and a one dollar cost for targeted information 
notes were reaonable costs in terms of the resources that 
could be returned by successful proposal funding. 
CoT yrmentarY 

Based upon the studies cited in the preceding 
paragraphs, it can be noted that there have been attempts 
made to ascertain both the kind of services .provided in 
support of proposal development, the quality of service, the 
preferences for services, and some idea of their costs 
within academic settings. In general, it would appear that 
these studies support a position that the services deemed 
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important focus upon the creative act of proposal writing 
and funding with a lesser emphasis on mechanical aspects of 
the process. In general, such assistance is sought from the 
immediate support area supplemented by remote offices. 
Training in proposal writing was deemed to be of some value 
as was the provision of information services. 

The general picture derived from the combined studies 
indicates that most organizations feel that support services 
are justified in relation to the potential benefits received 
j from the acquisition of external funds. There were, 

however, several studies identified which attempted to 
justify the costs of development in relation to the return 
on the investment. These studies are reviewed in the next 
section. 

IV. PREPARATION COST AND RETURN RELATIONSHIPS 
The preparation of proposals requires individual time 
and effort as well as the provision of organizational 
services to support such efforts. Thus, there are costs 
associated with the creation and submission of proposals. 
In view of the resources utilized, a question might be 
raised as to whether or not there is an adequate return for 
the investment involved. The purpose of this section is to 
review papers and reports seeking information on both the 
costs of preparation as well as examining the return on the 
investment . 

Seven studies were identified that sought information 
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on cost estimates. Three reports did not involve empirical 
approaches to estimating such costs but rather reflected 
-guesstimates" based upon an inductive analysis of 

preparation activities in selected situations.. These papers 
are reviewed here because they were considered forerunners 
to developing preparation costs estimates and are often 
referenced by other authors. The remaining four studies 
sought data relative to preparation costs and often matched 
them against funds received in order to secure some idea of 
return on investment. 
■ftiieBBt.ii natf»«" of Costs 

Chiappetta (1973) projected the costs of proposal 
development at Indiana University in response to a 
solicitation from the National Institute of Education. 
Dsing an estimated average for a professor's salary, time 
given to writing, clerical assistance provided, and related 
aspects, a cost of $2,035 for an individual proposal was 
secured. It was noted that 27 proposals were developed 
leading to a university investment of $54,035. A similar 
approach was taken by Buechner (1974) at Kent State 
University and with cost of proposal development estimated 
at $1,253 for the same program. He noted that if donated 
time was taken out of the total, the cost would be reduced 
by $1144 or an actual total cost of $190 dollars. He then 
notes that a total of 3,126 proposals were submitted for an 
aggregate investment of about 4 million dollars to spend 
11.3 million which was the final amount awarded. On this 
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basis, the author suggests that a minimum of 340,000 was 
invested to secure 11.3 million which he views as a good 
return on investment. 

While not attempting ■ to project a dollar figure, 
Leopold (1979) also estimated the amount of time and 
person-hours devoted to the preparation of proposals and 
questions the return on investment. He noted that in 1978 
there was an approximate total of 47,500 proposals submitted 
to five different federal agencies in one year. Using an 
estimate of three weeks as the average time to write a 
proposal, he estimated 2,700 man-years went into proposal 
development which he considered to be a conservative 
estimate. Adding 3 person-days to review a proposal for a 
total of 575 man-years, an estimated total of nearly 3,300 
man-years was given to proposal development and review. 
Considering the rejection rate running from 70 to 85 
percent, he questions the investment of time while believing 
that proposal creation will increase due to the competition 
for funds. The author suggests that the act of writing 
proposals to do research could serve to bring active 
research to a halt! 
Empiric al Cost-Return Studi es 

The above reports attempt to estimate the costs of 
proposal development but without the accumulated data from 
actual proposal development and relating such costs to the 
return on investment in terms <.>f dollars received. 

Teague and Heathington (1979) examined the costs 
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incurred by an academic unit at a higher education 
institution and then related this cost to funding secured. 
Development expenses were collected under six categories 
(staff time, typing, duplication, related costs, 

institutional routing, and indirect costs) for 71 proposal 
applications divided into 66 grants and 5 contracts. There 
were 65 new applications and 6 continuations developed for 
submission to different agencies with 46 applications going 
to federal agencies, 21 to state agencies, and 4 to private 
foundations* At the time of the study, 28 proposals were 
approved, 19 disapproved, and 24 were pending. 

The range of development costs was from $231 to $9,280 
with the average being $1,961. Personnel time was the 
largest cost ranging from $50 to $6,500 with an average of 
$1,124 and accounting for about 58 percent of average total 
cost. Based upon the proposals acrepted or rejected, the 
cost of preparation was $82,202 while the funded proposals 
received $1,072,542 for a return on investment of over 10 
times the cost. 

in view of the findings from the above study, Hensley, 
Gulley, and Eddleman (1980) did a study which recognized 
that proposal pay-off might not be independent of the 
institution involved and that the investment would vary from 
agency to agency. To secure a better idea of costs, their 
study divided the proposal development process into four 
phases: Matching Ideas and Funding Sources, Development of 
the Letter ot Inquiry, Development of the Prospectus or 
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Pre-proposal, and the Development of the Proposal and an 
Assessment for Project/Program Concurrence. Using a case 
study methodology, costs associated with the four phases 
were logged for four differer.- but related proposals being 
developed for submission to foundations, private business, 
state, and federal agencies. 

Average costs for the first phase were $360 dollars 
with private business costs highest at $616 and Federal 
lowest at $145. For the second phase, the average cost was 
$348 dollars with Federal costs lowest at $214 and state 
agency costs highest at $645. For the third phase, the 
average cost was $661 with highest being $1,289 for 
foundations and the lowest being Federal at $245. For the 
last phase, the average cost was $3,428 with the Federal 
costs being lowest at $2,042. The authors note that the 
costs summarized over the four phases for the Federal 
project were $2,554 but costs only for the last phase were 
close to the average reported by Teague and Heathington. 
Detailed data were presented for the one Federal project. 
The total amount approved for the project was $125,231 
giving about a 2 percent return on the investment. The 
authors note that the costs for the Federal project were 
much lower and therefore it was more advantageous for the 
university to develop proposals for that agency. No reasons 
were given as to why costs for private business and 
foundations were much higher on the average. Explanations 
as to the low cost of Federal projects focused upon the 
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careful definition of requirements and information by the 
funding agency, availability of priority area deadlines at a 
sufficient time before the deadline, and the well developed 
criteria for review which made proposal development an 
easier task.' The other areas were noted to be more 
subjective and required more personal interaction between 
the faculty and the agency program officer. Based upon 
their findings, the authors noted that the average costs 
were probably higher than previously reported and that there 
are variances in proposal development costs in terms 'of the 
sector from which support is sought. 

In contrast to the development costs associated with 
relatively large scale proposals are those associated with 
the development of relatively small grant proposals as 
reported' by Rogers, Sanders and Levenson (1970) in their 
review of the Small-Project Grants Program for the Bureau of 
Research, Office of Education (DHEW) . 

They report that the median time devoted to proposal 
preparation was about 48 hours. They also note that there 
was some relationship to funding since for those who devoted 
less than 20 hours time only 33 percent were funded while 
for those who spent between 41-60 hours there was a 41 
percent funding rate. When asked to estimate costs, the 
median was $48 with a range from $25 to $100. For over 70 
percent of the applicants, this cost was picked up by the 
institution while 20 percent personally paid the cost. 
Their data also indicated that the applicants paying their 
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own costs also had the lowest funding proportion. 
ft Non- Ac * domic Example 

The six reports noted above attempt to estimate or 
determine actual costs associated with proposal development 
in academic settings. One report (1971) was identified that 
attempted to summarize costs in connection with the 
development of proposals associated with aerospace 

corporations bidding on the procurement of a weapons 
systems, in this case the B-l bomber. The study was 
undertaken to establish a data base regarding proposal 
preparation costs, to determine factors associated with the 
costs, and to reduce costs. In May 1969, a Department of 
Defense decision was made to move ahead on the development 
of a prototype aircraft with the RFP being released in 
November. The plan called for the award of a systems 
integration contract to one of three airframe competitors, 
the engine development award to one of two engine companies, 
and an award to one of two avionics subcontractors of the 
airframe contractor. The original draft document for the 
airframe RFP called for 40 volumes but was reduced to 26. 
The final RFP page-1 irnited three volumes and imposed a 2,3 00 
limit on total pages for 13 of the other volumes. The 
Technical Proposal was to be submitted in 60 days, 
Management and Cost Plan in 98 days, and Trade Studies 
volume in 105 days. 

For cost collection, the proposal preparation period 
was from November 1969 to February 1970 and the_ Proposal 
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Evaluation/Source Selection period from February to June. 
For the three airframe and two engine companies, proposal 
preparation costs were $19,000,000 with 915 man-years of 
preparation with $9.5 million considered as not recoverable. 
For the evaluation period, the total costs were $16,700,000 
with 801 man-years and $8,900,000 not recoverable. Total 
costs were $35,800,000 with 1,716 mar\-years and $18,400,000 
not recoverable. If bid and proposal costs, independent of 
R and D costs plus other costs from 1964 to 1970 are 
included, the total is $65,600,000 for the 7 competitors. 
Combined with proposal development, the total was $101.3 
million. The results of this study in weapons systems 
procurement reinforces the point made in the study by 
Hensley, Gulley, and Eddleman that proposal development 
costs vary from institution and agency as well as the item 
being procured. It seems reasonable therefore that any 
attempt to project proposal development costs would be 
dependent upon such variables. Further, the task of 
collecting costs depends upon how the several phases 
associated with proposal development are categorized. 
Commentary 

In reviewing the several reports on cost estimates 
presented in this section, three main observations seem to 
emerge. First, that while the evidence is limited, 
estimates of costs based upon an intuitive approach do bear 
some similarity to costs based upon empirical data. It may 
well be that the intuitive approach draws from data based 
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upon experience which becomes more visible when actual data 
is collected over a set of specific proposals. Second, 
costs for proposal preparation and thus thei r relationship 
to a return on the investment is a consequence of several 
variables, among them the specificity of an RFP, the agency 
requirements from which support is sought, and the item 
being produced. Third, there seems to be a reasonable 
methodological approach to developing a cost-return ratio 
which leads to a justification that there is a sufficient 
positive ratio that organizations can justify the investment 
of resources to generate and secure proposal funding- In 
view of the one finding that costs- for submission to a 
Federal agency were lowest, there needs to be an additional 
study of this finding or else the higher costs of 
preparation for foundations and private business could 3erve 
as a deterrent to submitting program-related items to such 
agencies for possible support. 

V. REVIEWING AND EVALUATING PROPOSALS 
Once a proposal is developed and processed internally 
within an organization in the manner described by Jebens and 
others in the preceding section, it is then submitted to the 
external funding source for its review and funding decision. 
In the typical case, this review process involves a 
competition in that many proposals may be submitted to the 
agency but only a few are finally funded . 

The process of choosing among several competing 
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proposals falls under the more general concept of project 
selection. Like the literature on proposal development, 
there is voluminous literature, both of an empirical and 
non-empirical nature, on project selection a* it operates in 
a wide range of settings. The selection methods run the 
range from the development of mathematical models using 
"return on investment" (profit increase, market share, etc.) 
as a dependent variable and various evaluative criteria as 
predictor variables to other models placing almost complete 
reliance on subjective personal judgements albeit with some 
quantitative ratings being involved. Because of the large 
volume of literature on project selection methods and 
research, it was not viewed as being possible to include 
that totality in the current paper. Summaries of these 
methodologies, such as that prepared by Baker and Pound 
(1964) are recommended as good sources of information, 
instead, the focus here will be upon empirical studies which 
relate to various dimensions of the review and evaluation 
process as it operates principally within the natural and 
social-behavioral sciences in academic settings. 

Because of the importance of the process of reviewing 
and evaluating proposals, there have been quite a few 
studies relating to this area; perhaps more than to any 
other topic in the proposal development activity. Many of 
these studies have focused on identifying reasons for 
proposal acceptance and rejection, factors determining final 
decisions, the potential influence of personal factors in 
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peer review situations, and similar topics. An attempt has 
been made to categorize the studies so as to correspond to 
various steps in the review and evaluation process. In some 
cases, only one study is relevant to a section while others 
are included in several sub-sections. 
Tnitial Compl ian c.e .Screening 

In the typical case, proposals are initially reviewed 
at the funding source for general compliance with guidelines 
and relationship to program or agency objectives. Larsen 
(1975) conducted an investigation focusing upon the initial 
screening of grant applications sent to public foundations. 
The study involved interviews with 21 foundation managers 
plus a total of 25 hours of observation of board meetings 
and one session of a grantsmanship training program. In 
addition, archival items such as annual reports, form 
letters, and IRS reports were examined. Materials were 
collected from November 1973 thru April 1974 from 
foundations located largely in the San Francisco Bay area. 

The report noted that each foundation manager developed 
a personal classification system for reducing the number of 
submitted applications for subsequent review. Several 
different criteria were used ranging from a formal item such 
as presence of a certificate of IRS non-profit status to 
more informal means. Formal means included meeting of 
foundation priorities, geographical location, source of 
financial support, size of request, and organizational 
affiliation. Larsen noted that informal classification 
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systems were built on such concepts as "worthiness" and 
dependency". "Worthiness" had subclasses of personal, 
programmatic, and institutional. "Dependency" focused upon 
whether or not the agency seeking funds would become 
dependent upon a continuation of funding. If so, they were 
not likely to receive initial funding. The author noted 
that foundation managers attempted to control the behavior 
of their clientele to some degree using as a basis their own 
presumed knowledge and experience. An important dimension 
of this control, which was also related to worthiness and 
dependency, is that of determining "authenticity" which 
involves the seeking of information about clients to remove 
false fronts. For those interested in ascertaining on how 
foundations screen applications, the article would be 
informative. From a research methodology perspective, 
Larsen noted that participant observation was a much more 
valid method than telephone interviews in studying the 
activities of foundation managers. 
Formal wpviftw Process 

The most commonly used formal means for reviewing and 
evaluating proposals after the initial screening, 

particularly scientific research proposals, is the process 
referred to as "peer review". Over the years, knowledge 
gained from experience has established that the most 
preferred way to make necessary judgements about quality of 
a proposal is to have the proposed effort judged by persons 
who can be considered as an individual's peers on the 
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position they would be most knowledgeable about such 
dimensions as significance, purpose, method, and likelihood 
of results. Much has been written about the history, 
operation, varieties, and validity of the peer review 
process (Carter, 1974; Foster, 1976; Vandette, 1977). Some 
studies relating to these issues will be reviewed later in 
this report. 

Once screened, proposals are typically routed to a peer 
panel which independently rates them against evaluative 
criteria culminating in a summative rating and/or score. 
After the independent rating, the same panel members or a 
second echelon group may be assembled together to make final 
recommendations. In these face-to-face sessions, discussion 
focuses on individual proposals with each rater being given 
an opportunity to defend, explain, or even to change his or 
her initial rating. The exchange of information among peer 
panelists therefore would appear to be an important factor 
establishing the final ratings. 

p»at Tnformat-ion Exchange . One study was identified 
which attempted to examine the interaction among peer 
panelists. Paxton (1978) conducted a study, largely 
exploratory in nature, to determine the amount, type, 
frequency, substance, and related aspects of the information 
exchange among peer panel members. Drawing upon the studies 
of small group processes and direct observation methodology, 
an initial study was conducted to determine the information 
dimensions to be recorded and the process for so doing. A 
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revised form was developed to record information/comments 
relative to the substantive nature of the proposals, 
comments of a non-proposal but yet of substantive nature, 
and comments not related to eitl . . the proposal or general 
substance. A four member panel reviewing four ESEA Title 
III proposals in the state of Ohio was observed during a 
single rating period. Entries were made for each complete 
opinion or question made by a panel member. Each comment 
was given a point credit and then summed for the matrix for 
each panelist. 

The analysis of the individual matrices showed that one 
panelist consistenty had the most frequent comments. For 
the four panelists, most of the comments were related to the 
substantive nature of the proposal. The frequency of 
comments declined, however, with succeeding proposals being 
reviewed suggesting a fatigue factor. The frequency of 
comments did vary according to the final proposal rating but 
no consistent pattern was observed. 

In addition to the observation period, a self-report 
form sought perceptions from 48 panelists participating in 
the review process as to how the information exchange 
influenced their own rating behavior. Each panelist was 
asked to respond in terms of one proposal, identified by a 
title, from the set which they reviewed. No two panelists 
responded to the same proposal- The analysis of the 
responses indicated that more than half of the panelists did 
change their initial rating after hearing information 
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presented by other panelists while about one-third did not 
change. In terms of reasons for changing, the two most 
frequently given were perceived expertness of a panel member 
and the quality of the information presented. When 
questionable proposals had follow-up sessions for further 
information, only half of the panelists change their rating 
with this additional information. The panelists not 

changing their initial rating felt that there was no new 
information presented in the small group session. The 
panelists did report a feeling that the session was 
dominated by a group member but this did not relate to their 
final ratings. 

Paxton concluded that it was feasible to study the 
information exchange in peer panel meetings; that the 
quality of information exchanged was of sufficient quality 
and quantity to provide for reasonable ratings by panelists; 
that information exchange did influence some changes in 
ratings ; and that most reviewers expressed beliefs that 
their ratings were not affected by the time of day (sequence 
of rating) or the dominance by one or two group members. 

A greement Among Panelists . The Paxton study suggested 
that there was an eventual agreement among the raters 
regarding the proposal reviewed. The relative nature of 
such agreement, however, could be high or low. Miner and 
McDonald (1981) conducted a study to establish the relative 
agreement among a set of raters by investigating the 
reliability of ratings on proposals submitted to an internal 
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faculty grants program at Marquette University. The 
proposals involved summer fellowships which included salary 
and regular grants which did not. 

in 1979, 52 summer grants were submitted with 20 funded 
while 43 regular proposals were submitted with 24 awarded. 
Each proposal was rated on a seven point scale using 
administrative guidelines. Scale values were summed and 
averaged. The average score counted for 50 percent of total 
rating with the other 50 percent derived from a pair-rating 
system using two experts who met and determined a joint 
rating score. The same procedure was used for both types of 
grants. Reliability was established using an intra-class 
correlation coefficient. For summer grants, the reliability 
was +.68 and for regular grants +.59. The authors concluded 
that there was little inter-rater reliability. Analysis of 
the scale items revealed difficulty with mid-range 
proposals. 

Based upon the results, three changes in procedure were 
made. The committee was changed from 8 to 10 members; a 
reviewer's checklist adopted; and the weighting changed to 
67 percent for a single rating and 33 percent for the paired 
rating. This new procedure was used in 1980 for 54 summer 
applications with 19 awards and 34 regular applications with 
22 awards. Intra-class correlations for summer grants were 
+.85 and for regular grants +.79. This same procedure was 
used again in 1981 with 33 summer applications and 51 awards 
and with 35 regular applications and 25 awards. The 
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resulting intra-class correlations were +.89 for summer and 
+.80 for regular grants. The authors concluded that the 
revised process led to a greater stability or agreement 
among raters and is now the current procedure. 

While the Paxton study and the work of Miner and 
McDonald involved the agreement among panelists in 

evaluating proposals, they did not examine the results using 
different groups of panelists. One study making a 

comparison of the agreement from two different groups of 
panelists was that done by Foster (1976). The items 
reviewed were preliminary grant proposals submitted to the 
Comprehensive Program of the Fund for the Improvement or 
Post-Secondary Education in 1975. Two groups of readers, 
one identified as F_ifild Efiadfii group and the other as F i e l d 
Advisor group, rated the proposals. The Field Reader group 
involved 74 individuals from various post-secondary 
education agencies who had not submitted a proposal to the 
program plus FIPSE staff. The Field Advisor group consisted 
of 1,077 volunteer applicant-reviewers who agreed to have 
their own proposal reviewed by other applicants and who also 
agreed to review five proposals each. In contrast, the 
Field Reader group was to review 20 to 25 preliminary 
proposals on a voluntary or fee tvasis. All reviewers were 
provided with forms and also completed a questionnaire 
seeking profile data and opinions about the field advisor 
By st era . 

Each preliminary proposal was rated as In, Possibly In, 
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Possibly Out, and Out. Of the group rated as In and 
Possibly In, the ratings were reviewed by FIPSE staff and a 
final set of applicants asked to submit detailed proposals. 
Of the 1,850 preliminary proposals, 180 detailed proposals 
were requested. The detailed proposals were reviewed by the 
FIPSE staff and funding decisions made. 

Comparisons of rating performance and agreement between 
the two experimental groups (Field Advisor, Field Reader) 
and the control group (FIPSE staff) were made by use of mean 
scores and correlations with the author reporting both "t" 
values and correlations coefficients. The analysis in terms 
of the question about the agreement between the two 
experimental groups (Advisors and Readers) revealed a 
correlation of .25 for all preliminary proposals, a 
correlation .07 for proposals In and Possibly In, and 
correlation of .23 for Possibly Out and Out proposals, and 
.11 for In proposals. Two of the corralations (.25 and .23) 
were significant at the .01 level. The highest significant 
correlation (.01 level) was .54 between the FIPSE Staff and 
Field Reader reviews for all preliminary proposals, followed 
by a significant correlation of .41 between these same two 
groups for Possibly Out and Out proposals, and .38 for the 
Staff-Reader agreement. 

Based upon these correlations and the results of the 
tests of means, the author concluded by noting that the two 
different groups gave reviews that are comparable. A 
notation is made that the two groups, Advisors and Readers, 

43 



Page 40 



9 

ERIC 



had quite similar profiles and in terms of position and 
institutional locations thus negating these as factors. In 
terms of opinions about the field advisor system, 70 percent 
of the Advisor group thought the procedure was more 
beneficial than the Field Reader approach while 18 percent 
thought it was worse. The prime reasons for feeling it was 
not as beneficial related to possible conflict of interest 
and unknown competency of the Advisor group. 
Comparisons Between Accepted and Rejec ted Proposals . 

Once funding decisions have been made, interest focuses 
upon why some proposals were winners and others losers. 
Were rejections and awards based upon favoritism, politics, 
panel bias, inadequate review processes, or a host of other 
possible explanations or rationalizations? Most reference 
texts dealing with proposal development cite the early study 
of Allen (1960) on deficiencies of 605 Public Health 
Services grants, the work of Smith (1964) in citing the 
weaknesses of proposals rejected by the Cooperative Research 
Program of the D. S. Office of Education, and/or the study 
by Chalfant and Nitzman (1965) on limitations of 

applications to handicapped children research program. 
These studies have not been reviewed in this report because 
they are so commonly referenced and the fact that they 
report only lists of deficiencies or inadequacies and do not 
involve comparisons between accepted and rejected proposals. 
The studies cited here involved some form of comparison in 
one manner or another. Two studies were identified that 
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made compar isor.3 on selected proposal characteristics as 
they relate to ratings on evaluative criteria or funding 
decisions. Interestingly both were doctoral dissertations 
using similar statistical analysis procedures in focusing 

upon the same issue. 

The relationship between the presence or absence of 
selected proposal characteristics to their total rating was 
examined by Wilson (1977) for 353 applications out of 361 
submitted to the Vocational Education Research Branch of the 
Office of Education. Proposal attributes intuitively 

selected based upon empirical definition, mutual 

exclusivity, independence from the dependent variable, and 
apparent variability across proposals were used as 

independent variables. The characteristics included such 
items as table of contents, abstract, review of literature, 
tables and charts, identification of project director by 
name, citation of references as footnotes, and glossary of 
terms for a total of 22 such characteristics. The dependent 
variable was the mean score computed for total score ratings 
summed over evaluators. The mean scores for proposals 
having or not having the attribute present were tested by a 
one-way analysis of variance. Multiple regression 

techniques were also used in order to establish the unique 
contribution of each attribute. 

The analysis of each attribute showed that only the 
presence or absence of the literature review was not 
statistically significant. In addition to the application 
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characteristics, the multiple regression included the 
priority area addressed by the proposal and the type of 
institution making application. The multiple correlation 
(R-squared) was +.463. In terms of stepwise procedure, the 
item making the greatest contribution was the priority area 
of application followed by the agency making application, 
and the several proposal characteristics. 

In contrast to the Wilson study which looked at a wide 
range of general proposal characteristics, Howarth (1980) 
investigated the technical factors or research methodology 
variables which would serve to discriminate between good and 
poor proposals. The proposals involved were those submitted 
and funded in connection with the National Institute of 
Occupational Health and Safety and drawn from the 
Cincinnati, Ohio area. 

Osing research literature, a list of 63 technical 
aspects of research design and analysis was establishea. 
Forty-four project officers for 72 projects currently active 
in 1979 or within three months of completion were asked in 
an interview to indicate relevancy (or irrelevancy, 
optional, or uncomfortable) of each variable for the 
research project involved. The frequency of relevance for 
each of the 63 characteristics was presented with "Relevant 
Experience" and "Availability of Personnel." being the only 
items identified 100 percent of the respondents. 

A second phase of the study involved reviewing the 
rating sheets for 13 evaluative criteria used in connection 
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with proposals submitted leading to funded projects. Each 
of the 72 contracts originally had 1 to 13 proposals (author 
is inconsistent on the latter number) submitted to it. The 
dependent variable used in analysis was the total points 
assigned out of a possible total for all criteria, A total 
of 240 proposals, 125 accepted and 115 rejected, were used 
in the analysis. These proposals represented 48 of the 72 
contracts. Discriminant analysis and t-tests were done on 
each evaluative criteria. A step-wise multiple regression 
was also done on the evaluative criteria. The results of 
the t-test for the 13 evaluative criteria showed the 
differences were all statistically significant except for 
Information Sources and Past Performance, The stated 
results are at variance with tabled results which show that 
Data Collection has a value of .604 and Past Performance 
with a £ value of .000. The results from the discriminant 
analysis were presented in terms of highest overall 
predictability of group membership for each individual 
evaluative criterion. On this basis, Experience with a 
Particular Agent has the greatest predictability with 80 
percent of acceptable proposals predicted correctly and 71 
percent of unacceptable predicted correctly. Items such as 
Study Site Selection, Sampling and Analysis, Past 

Performance, Information Sources, and Data Collection were 
in the 50 percent area of predictability. It should be 
noted that in many cases the discrimination was based upon 
comparing the accepted and rejected proposals numbers even 
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though number of proposals rated on each of the criteria 
varied from 113 to 236. 

In terms of the regression analysis, results are 
presented showing that the R-square ran from a +.38 with one 
variable, Technical Merit of the Approach, to +.76 with the 
addition of the last evaluative criterion of Past 
Performance. 

In summarizing the findings, Howarth notes that of the 
13 evaluative criteria, nine predicted correct group 
membership at levels from 98 percent for Plans for Data 
Collection Analysis to 76 percent for Availability of 
Proposed Staff. Because of the variable number of proposals 
rated on the several evaluative criteria, she noted that 
more reliance can be placed on the Specific Experience 
criterion since it was rated on 236 proposals than on the 
Data Collection criterion since it was rated only on a few 
proposals. Three criteria were considered as poor 
discriminators; Thoroughness of the Proposed Approach, 
Technical Competence in Sampling, and Past Performance. In 
turn, these predicted unacceptable proposals at a high level 
of membership- Based upon the results of the discriminant 
analysis and the regression data, the author believed that 
the results were generalizable to other contract proposals 
employing similar evaluative criteria. 
A pplic ant Feedback. 

Once a proposal has been submitted, reviewed, 
evaluated, and a funding decision made, applicants are 
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notified of acceptance or rejection. Typically, winners do 
not seek information or feedback about their proposals but 
losers desire to know on what grounds or basis their 
proposal was rejected. Two studies were identified that 
relate to the feedback situation and are the subject of this 
section. 

To examine the nature of the feedback provided by 
funding agencies to rejected applicants, Beathington and 
Teague (1980) solicited information from 127 faculty members 
in higher education insitutions. The faculty members were 
from the areas of health science, engineering, education, 
and business who had proposals rejected by NlH, NSF, OE, 
NIE, National Endowment for the Humanities, and National 
Endowment for the Arts in fiscal 1979. Responses were 
obtained using a 16 item questionnaire. Of the group 
surveyed, 117 or 92 percent had previously submitted a 
proposal with the average being 5 submissions. The group of 
127 averaged two federal grant awards with 28 percent having 
not received a previous award. 

Follow-up requests for information regarding their 
proposal were made by 74 percent of the applicants. Fifty 
telephoned, 38 sent letters, and 6 arranged for personal 
meetings. Reviewer worksheets were sent to 54 persons, an 
agency summary sent to 30 others, and 8 talked personally. 
Fifty-seven percent of the respondents received responses 
within two weeks and an additional 33 percent received 
responses in four weeks. 
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While no data is presented with regard to frequency, 
the authors noted that the rejected applicants' reactions, 
after examining the reviews, clustered around five major 
reactions: the differences among reviewers, the differences 
between review panels, non-constructive or overly general 
comments, the qualifications or credentials of the 

reviewers, and the possibility of a closed network or 
in-group. Recommendations for improving the feedback 
process were presented. 

The prior report focused primarily upon reactions of 
rejected applicants. The study by Rogers and others on 
small grants cited earlier collected data with regard to the 
review and feedback process from both applicants and 
reviewers. Applicants' responses were gathered on such 
items as length of time to decisions, how they were 
notified, reasons for rejection, and availability of 
field/reader comments. Of the total applicants, 40 percent 
thought the funding decision took Considerably Longer than 
expected, 27 percent Somewhat Longer, 29 percent saying 
About What Was Expected, and only 5 percent saying Less than 
Expected. Applicants not funded were asked" if they 
requested an explanation of the decision and 71 percent 
indicated they had. The response most frequently given was 
Poor Design (39 percent) and Lack of Educational 

Significance (32 percent) with other aspects receiving 
lesser amounts (7 to 4 percent). No explanation was 
provided to 21 percent of the not funded applicants. When 
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asked if the field reader comments should be sent routinely 
to each applicant, approximately 85 percent of both funded 
and nonfunded applicants felt they should. The field 
readers also supported this idea but the percentage was 
lower with only 59 percent indicating they should go to 
every applicant. In terms of satisfaction with the 
explanation, 81 percent of the non-funded applicants seeking 
an explanation checked n Not Satisfied." Respondents were 
asked about the criteria used to judge their proposal and 
577 or 87 percent indicated that the criteria were 
appropriate. 

The survey form completed by the field readers 
solicited information relative to a series of demographic 
items such as major field, distribution of professional 
time, number of proposals submitted, and number of 
dissertations directed. Of the total responses, 339 or 80 
percent indicated that they were under contract as field 
readers, the modal number of years for being a field reader 
was 3, and the most frequent number of proposals reviewed as 
6 to 14. The mean time for reviewing a proposal as 
individual field reader was 3.49 hours but when reviewing to 
get ready for a panel meeting the mean time was 1.72 hours. 
In view of the relative size of the proposals, the field 
readers indicated that on the average about 15 proposals 
would be an optimum number for a panel to review in one day. 
Field readers were also asked about the nature of the RRP 
and their suggestions for improvement of the program. The 
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reader is referred to the study for details on these related 
items - 

In discussing the findings with regard to the proposal 
review, decision and feedback process, the authors noted 
that there were administrative problems such as budget 
freezes which created difficulties for rapid processing. 
The authors noted also that the program never had what might 
be called a "typical year" and the utility of the results of 
the small grant study are limited for that reason. 
Commentary 

The studies cited in this section have focused upon the 
process of selecting a set of proposals for funding from a 
larger group submitted. In contrast to possible 

organizational settings which utilize more objective models 
for selection, the settings reported here are primarily ones 
involving a largely subjective judgemental process wherein 
peers are used to rate proposals against a given set of 
criteria. It was interesting to note that no study was 
identified that sought to establish evidence about the 
validity of the criteria or their relative weighting in 
terms of a total. Some evidence was obtained that informal 
as well as formal initial screening procedures were used. 
Investigation of the actual process of making judgements in 
real-time was examined and indicated that the information 
exchanged among the members of a peer panel as well as who 
offered the information had some influence upon the final 
ranking of a proposal. To further assist in making 
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necessary judgements, comparisons were made between accepted 
and rejected proposals with various elements being examined 
to see if a consistent set of elements could be established. 
Only limited study has been made of the important feature of 
the review process of providing feedback to applicants as to 
reasons for rejection. In some instances, no such feedback 
was provided while in other instances helpful information 
was presented. In view cf the observation that judgement of 
individuals plays a major role in making final funding 
decisions, it would seem apparent that investigations should 
be made upon the factors relating to the judgements being 
made. The next section examines some of the studies and 
reports which have questioned the credibility of the peer 
review system, particularly within academic settings. 

VI. ESTABLISHING CREDIBILITY OF THE PEER REVIEW PROCESS 

While one cannot state with absolute assurance, it is 
quite likely that the general model of peer review for 
proposal evaluation has its origins in the colleagual 
evaluation system employed in academic settings for making 
judgements about promotion and tenure, the selection of 
research reports for publication in scientific journals, and 
similar situations requiring the use of expert judgements in 
specialized disciplinary areas. 

The process, while generally accepted, has not been 
without question. Several studies were identified that 
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report on the credibility of the peer review operation. One 
set of studies has focused primarily upon the peer review as 
it operates in Federal funding agencies. A second set has 
focused upon an examination of factors or variables which 
have been investigated as a response to charges that the 
system is biased. These two sets of studies comprise this 
section of the report. 
Intra-Federal Agency Studies 

The use of peers as the principal means of evaluating 
the scientific or technical merit of proposals has been 
employed by Federal funding agencies since the turn of the 
century (NIH, 1978). Vandette (1977) and Carter (1974) have 
reviewed and traced the numerous Congressional hearings with 
regard to the operation of the process and the validity of 
its judgements. These studies have been initiated because 
of such charges as favoritism/ cronyism/ "old boy" networks, 
and political influence (Gross, 1976). Three studies 
involving peer review operations within Federal agencies 
were identified. The agencies involved were the National 
Institute of Health, National Science Foundation, and 
Department of Education. Each report is summarized below. 

National Institute of Health Study . Because of the 
important role that peer review has in the support of its 
extramural research funding, the National Institute of 
Health in 1975 established an internal NIH Grants Peer 
Review Study Team. The Study Team was charged with 
examining the current system, exploring alternatives, and 
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making recommendations for any changes. The results of the 
Study Team's efforts are contained in two documents in the 
form of reports to the Director of NIH. The first document 
titled Phase I (NIH , 1976) consists of three volumes. 
Volume 1 summarizes the principal results of the study along 
with recommendations. Volume 2 contains a variety of 
background materials. Volume 3 consists of supplemental 
material relating to the preliminary analysis of data 
collected. The Phase II report (NIH, 1978) presents a more 
detailed analysis of the data collection and is viewed as a 
support document for the Volume 1 of the Phase I report 
which is considered the major document produced by the Study 
Team. Comments presented here have been selected from both 
sources . 

As noted, the initial effort to study the NIH peer 
review system started in 1975. The basic process used by 
NIH is two-tiered; an Internal Review Group initially judges 
the scientific significance and technical merit of a 
proposal and assigns it a technical merit priority score. 
The National Advisory Councils review the technical merit 
recommendations and make final reviews for scientific merit. 
The Councils also make recommendations as to program 
relevancy and funding priority. Three major means of 
securing perceptions about the peer review system were used 
by the Study Team. One involved a survey of the 1975-76 
review groups, a second involved a series of hearings, and a 
third was a solicitation of letters. 
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The survey form or questionnaire of review group 
members consisted of a section on demographic data, followed 
by a section on the assessment of the current system, one on 
the impact of recent and future changes in the system, and a 
section on suggestions for improvement. In 1 976 , the Study 
Team distributed 1,354 questionnaires to 12 Advisory 
Councils, 51 Internal Review Groups of the Division of 
Research Grants, and 24 Institute Internal Review Groups. 
In addition, the survey was also given to liaison members 
representing federal agencies and &d frQC consultants (both 
representing 13 percent of the total survey group). The 
overall response rate was 94 percent. No survey forms were 
sent to applicants. 

Respondents were asked to rate aspects of the current 
system from Excellent to No Opinion. The results of the 
analysis are presented in terms of the percent of the 
persons responding. Each item had a focused stem followed 
by a series of specific items related to the stem. The 
results are presented in tabular form organized so that the 
general specific items receiving the largest percentages are 
presented first followed by lesser percentages. Further, 
the items have been grouped into major percentage categories 
(e.g., 90 percent or more responding Excellent/Good). On 
this basis, the Phase I report highlights the following 
items as having the strongest endorsement: 

- Lack of general bias 

- Lack of bias against minorities, young 
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investigators , or women in recent years 

- Overall adequacy of current review in 
general for traditional research grants 

- Adequacy of the current review system for 
scientific and technical quality of new 
grants and the capability of research 
investigators 

- Value and quality of site visits in the 
review process 

- Performance of peer groups in discussion 
of applications and their behavior during 
the review process 

- Scientific and technical members qualifications 
and pe rf o rmance 

- NIK staff qualifications and performance in 
administering the system 

Those statements being viewed as haviia weak endorsement 
(those having 80 percent or less of all review group 
members) were as follows : 

- Some bias towards "cronyism" 

- The review of prc 3 ram project and center 
grants was judged less adequate than 
traditional individual research projects review 

- Adequacy of review for program relevance was 
judged Excellent or Good by only 67 percent 

- Reviews for budget appropriateness and 
essential collaborative arrangements were 
judged more favorably by Initial Review Groups 
than by the National Advisory Councils 

- Time available for site visits was least 
favorably rated by the IRGs who are 
responsible for such visits 

- The priority score ranking system apparently 
posed problems of understanding 

- Current restrictions on applicant notification 
has sizeable opposition 
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- Time available for review appeared to be an 
area of some dissatisfaction 

- Working conditions for Study groups were 
viewed less favorable by DRG groups than by 
other review groups 

- Selection process for peer review gyoup 
members was not heartily endorsed since less 
than three-fourths view it as Excellent/Good. 

- Public members 1 performance was rated Excellent 
or Good by about two-thirds of the National 
Advisory Councils 

The report highlights two items having the least support. 

Both were related to applicant notification. Only 56 

percent found favorable the current requirement prohibiting 

informing the applicant of the priority score from the 

Initial Review Group. Only 69 percent found favorable the 

requirement of delaying the notification of the overal 

Initial Review Group recommendation until the final review 

by the National Advisory Councils. 

In addition to the survey, the Study Team conducted 

three hearings around the country ? id also solicited letters 

from 30,000 interested parties. A total of 1,400 persons 

wrote letters and 93 persons presented oral or written 

testimony at the hearings. The Phase II report notes that 

the characteristics of the correspondents differed from the 

witnesses in almost all dimensions of obtained information. 

Witnesses were more often from formal organizations and 

included a higher proportion of women and individuals who 

had never applied for a grant. Correspondents were 

primarily faculty from higher education institutions. The 
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data suggested, as the Phase II report noted, that the 
witnesses were less knowledgeable about the system and less 
successful as applicants than were the correspondents. The 
comments presented by the witnesses and correspondents were 
analyzed by identifying 12,065 comments, classifying them 
into 106 topics, then into 64 categories, and finally into 
11 major subjects. The summary of strong points noted by 
correspondents and witnesses showed that 83 percent of the 
correspondents approved of the system while 73 percent of 
the witnesses approved of the systems. About 14 percent 
made comments about the presence or absence of bias 
indicating they felt the system was not biased in general or 
unbiased towards women and minorities. Notation is made in 
the Phase II report that the responses from the letters and 
hearings do not represent a scientific sample while the 
survey results are considered more valid in that they were 
considered as a representative sample. 

While the two reports contain recommendations for 
improvement of the system, overall the three groups of 
persons involved in the survey (reviewers, correspondents, 
and witnesses) , generally view the current peer review 
system as a satisfactory and reasonable way to evaluate 
grant applications. It was noted in the report, however, 
that individuals with the most experience with the system 
were more favorably inclined and that the grant review 
groups were more favorable toward the current system than 
were the witnesses and correspondents. 
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National Science Fo nnriat-inn Study. A second study 
examining the general overall function of the peer review 
process as a vehicle for proposal evaluation was that 
conducted by Vandette (1977) using the National Science 
Foundation " operations as the data source. The focus of the 
research was upon the "agency-to-individual " as contrasted 
to the "agency-to-institution" mechanism since the former 
makes use of either an individual or peer review panel in 
making final decisions. Four questions directing the 
research sought to determine whether or not the peer review 
system provided for the advancement of science in the most 
effective way, if the process was fair and impartial or 
subject to political influence and geographical favoritism, 
if it was economically feasible, and if it promoted 
"grantsmanship" and is too secretive. 

Testimony from oversight hearings held by the 
Congressional Subcommittee on science, Research, and 
Technology in 1975, a study of past trends and policies and 
practices with regard to the award of NSF grants, plus 16 
personal interviews with different government and 

educational sources including NSF serve as data sources. 
The interviews consisted of 3 persons from NSF, 6 from the 
National Institute of Education, 6 peer reviewers, plus 1 
person from the National Association of State Universities 
and Land Grant Colleges. Interviews were both taped and 
written. 

In summarizing the results, Vandette notes that while 
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the peer review system has faults no method superior to it 
has been found for judging the competence of proposals and 
that its positive aspects should be enhanced. In terms of 
promoting the advancement of science, the author concluded 
that the peer review system could do more in seeking out and 
supporting innovative research. In responding to the 
question of fairness, the author concluded that -here was 
perhaps some truth to the charge that there is some 
geographical/institution bias with regard to sources of peer 
reviewers. Tables are presented 'in the text showing the 
distribution of reviewers by geographical regions, 

institutional sources, and publication rates* The author 
noted that patterns cf funding in NSF tend to give a strong 
advantage to prestige institutions. In discussing this 
observation and the possibility of such a circumstance 
occurring, the author raised a question of scientific merit 
versus equity. He also noted that there is probably no real 
way to satisfy the critics of the system on this point. No 
real conclusion was drawn on the questions relating to the 
cost of the peer review system although the results seem to 
suggest that it is justified. In terms of opening up the 
system (e.g., making the names of reviewers public), the 
author concluded that the system would be harmed by such an 
action. He notes that the Congress itself in the NSF 
hearings suggested going slow on this type of action. In a 
final note, Vandette feels that confidence in the peer 
review system clearly exists and, while not per:ect, it is 
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the most feasible system devised. 

np parf.ment of Education Study , In response to requests 
from committees of the House of Representatives, the General 
Accounting Office did a review of the procedures used to 
award discretionary grants or. selected programs (GAO, 1983). 
The GAO was asked to secure information on the legislation 
and related items that governed the .grant award process, the 
establishment of funding priorities, the recruitment and 
selection of field readers, reader selection criteria, 
reader training and orientation, procedures for reviewing 
and ranking applications, differences between reader 
rankings and final selections, procedures used to determine 
final grant amounts, and percent of requested funds received 
in 1981 and 1982. In addition, the GAO was to compare 1981 
and 1982 competitions for selected programs with special 
emphasis on the composition of field readers. 

The GAO report examined three program activities:, the 
Women's Educational Equity Act Program, the Unsolicited 
Program of the National Institute of Education and a set of 
three programs under Talent Search. Details are presented 
regarding the process of awarding grants along with the 
selection and composition of field readers for each program. 
For purposes of this report, attention will be given mainly 
to the operations relating to the selection of field readers 
as they operated within each program. 

In reviewing WEEAP field reader selection, it was noted 
that in 1981 an informal and unsystematic process was 
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employed. In 1982, the program used the Field Reader 
Outreach Program of the ED in addition to its own 
procedures. For 1981, there was about 300 field readers 
while in 1982 there was a potential pool of about 400. 
There was a concern chat continued use of the same field 
readers year after year resulted in a "liberal" bias and 
that the use of the Outreach program would provide readers 
with a more "conservative" philosophy. The report presents 
information comparing the 84 field readers used in 1981 to 
the 55 used in 1982 on sex, race, educational level, area of 
residence, and place of employment. The analysis showed 
that there were significant (sic) differences in terms of 
ethnicity, area of residence and employment. In 1981, 80 
percent of the readers were Black, Hispanic, Asian American 
or Native American while in 1982 only 24 percent were from 
these groups. In 1982, more readers were from the Southeast 
and Midwest than in 1981. In 1982, there was a decrease in 
percent of readers from non-profit organizations and an 
increase in percent of unemployed and privately and 
self-employed persons. The report also notes that based 
upon a review of resumes, 1 of the 1981 readers and 11 of 
the 1982 readers did not meet selection criteria. In terms 
of sex, the percent of women was 86 for 1981 and 87 for 
1982. As for the awarding of funds based upon ranks 
resulting from the field reader reviews, the GAO report 
states that in 1981 the WEEAP staff selected applications 
for funding based on the ranks as well as additional 
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decision criteria. This resulted in applications for most 
priorities areas not necessarily being funded in rank order* 
In 1982, the awards funded the applications in their rank 
order. The report details the fact that in 1982 the 
selection of field readers was done by an Acting Assistant 
Secretary for Elementary and Secondary Education in the 
absence of the WEEAP Director. This condition could have 
resulted in both the contrasts in the 1981 and 1982 field 
reader composition and the awarding of grants in 1982 by 
rank directly. 

In reviewing the National Institute of Education, the 
report notes that in 1981 the reviewers were selected by the 
staff from the program areas involved. In 1982, the program 
areas were directed to select part of the readers from a 
list compiled by the director of NIE. The field reader 
groups for 1981 and 1982 were compared on the basic of sex, 
race/ethnicity, and knowledge of educational research using 
randomly selected information sheets from each reader. Of 
the 50 out of 205 reviewers used in 1981, 60 percent were 
White while in the 1982 group of 272 reviewers, 75 percent 
of the 60 files examined show White ethnicity. As for sex, 
the group from 1981 showed 54 percent male vhile it was 65 
percent male in 1982. In examining credentials for an 
understanding of educational research, it was noted that 
there was not sufficient information to permit a 

determination of research competency for 53 of the 205 
reviewers in 1981 and for 32 of the 272 readers in 1982. As 
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for funding based upon rank order, five proposals were 
funded in 1981 but not as in the final rank order. In 1982, 
there was some variation by program area but in the 
unsolicited proposal case, 13 proposals were selected for 
funding that were not part of the top 17 ranked. Deviation 
from the final order was justified on the ground that 2 
addressed an ED priority, 5 supported the NIE mission, and 9 
offered unique research opportunities. 

Examination of the Talent Search procedures noted that 
the field reader selection was done by randomly selecting 
200 readers from a file of persons identified as qualified 
to- read applications for Talent Search. Comparisons were 
made between the 1980 and 1982 groups on the basis of sex 
and ethnicity. In 1980, there was 69 percent male while in 
1982 it was 53 percent males. In 1980 there was 57 percent 
White and in 1982 there was 32 percent White with Blacks 
showing an increase from 29 to 45 percent. Of the total set 
of 268 ranked projects, 159 were recommended for funding and 
these were essentially as in rank order. Due to subsequent 
availability of funds, lower ranked projects were also 
funded. 

The report does not present any conclusions about the 
relationship between field readers, their recruitment and 
selection, and the process of grant awards. It presents the 
findings above as fact leaving the reader to draw his or her 
own conclusion. For purposes here, the findings suggest 
that the compositon of peer panels in various programs of 
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the Department of Education has diversity in how they are 
selected and their demographic characteristics. There is 
also variation in the manner in which the results of the 
field reader rankings are related to the making of final 
awards. The findings also note that factors other than the 
public evaluation criteria are used in making final awards 
after rankings are obtained from the field readers. 
Studies Relating to Potential Peer Bias . 

Even though the internal studies present evidence about 
the credibility of peer review, charges about bias in the 
award process do exist. Four studies were identified that 
were aimed at substantiating or ruling invalid such charges. 

Liebert (1976), using data from a 1972-73 American 
Council on Education survey of 259 senior colleges, examined 
determinants of grant-getting on a national basis. A total 
of 5,687 individuals, or a 15 percent subsample was drawn 
from 40,421 responses. This total was reduced by 
eliminating those with the highest degree earned in the last 
two years or where there was no data on grants or 
productivity leaving a balance of 4,949 cases. An item 
asking about the number of agencies from which grants were 
secured as a measure of grant-getting plus two productivity 
items on the ACE form relating to total number of published 
articles and manuscripts published or accepted in the last 
two years were used as independent variables. 

Using path and regression analyses, the author noted 
that other than field and productivity variables not much 
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else made any difference. He noted also that the weak 
relationships did not support claims of institutionalism and 
the need to have agency contacts. In summary, Liebert found 
that the distribution of research g rants was more 
competitive with regard to individual productivity criteria 
than it was biased by field favor. There was little 
evidence of situational or personal particularism in the 
sample studied. 

In connection with the analysis in 1975 of vocational 
education proposal awards, Wilson (1976) analyzed the 
relationship of selected rater characteristics to proposal 
ratings. The specific characteristics investigated were 
sex, ethnic group membership, highest degree earned, field 
of degree, and place of employment (Office of Education, 
other f ede ral agency , educational agencies, and 

non-educational agencies). A total of 29 raters were 
involved in the study. Mean scores for each rater over all 
proposals they rated were determined along with mean ratings 
for subsections of the proposals. One-way analysis of 
variance was employed on each characteristic. 

There was a significant difference on the rater ethnic 
group membership at the .10 level with American Indians 
giving the highest mean ratings followed by Whites, Blacks, 
and Hispanics in that order. Differences by earned degree 
were significant at the .05 level with MeD degrees having 
the highest mean rating and the PhD group having the lowest 
average rating. In terms of employment location, there was 
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a significant difference between Office of Education and 
non-Office of Education employees at the .05 level with OE 
personnel having a higher mean rating. Office of Education 
raters also had significantly higher means at the .05 level 
than did raters from other Federal agencies. There were no 
differences for the characteristics of sex, field of degree, 
and employment in educational or non-educational agencies. 
While not true for all sections, the significance of mean 
ratings for subsections of the proposal tended to be 
correlative with the overall means ratings significance. 

A third study relating to potential influencing 
potential factors operating in the peer review process was 
conducted by Ormiston (1977) in the field of education. The 
particular proposals of interest here were those submitted 
to rhe Basic Institutional Development Program of Title III 
of the Higher Education Act of 1965. To develop a 
background for the study, the author secured information 
about peer reviewers for tr.e period 1968 to 1976. The 
fiscal year 1975 was selected to study in depth the 
relationships that might exist between reviewer ratings and 
institutional characteristics associated with the reviewers. 
In 1975, three panels at three time periods rated proposals 
on a 1 to 5 basis. A total of 58 peer panel reviewers were 
grouped according to institutional level (2 or 4 year), 
source of institutional control (public or private), and 
minority status (predominantly white or black enrollment) . 
A separate group of 12 reviewers was classified on place of 
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employment (education— other nongovernment agencies) . The 
480 applicants were also categorized on the same first three 
variables. A total of 18 questions guided the collection of 
data with Chi-square being the test statistic. In addition 
to ratings, data regarding funding recommendations was also 
obtained. 

Findings for each of the 18 questions were presented in 
tabular form. As presented, no indication was given 
regarding the results of the Chi-square test leaving the 
reader to the conclusion that there was no significant 
relationship for each of the questions. The findings with 
regard to institutional level of the reviewer indicated no 
relationship between assigned reviewer ratings and level of 
institution being evaluated. There was an observed 

relationship in th* : two year institution reviewers tended 
to favor two year institutions while four year institution 
reviewers also tended to favor two year institutions. On 
the variable of public or private control, no relationship 
was observed between reviewer rating and control of 
institution being evaluated. There was a relationship in 
terms of recommendations for funding in that reviewers from 
private institutions tended to favor public institution- in 
their recommendations. As for the minority factor, there 
was an observed relationship between reviewer ratings and 
institution evaluated. Both white and black reviewers 
tended to give higher ratings to black institutions. In 
terms of funding recommendations, both reviewers from white 
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and black institutions tended to favor predominantly white 
institutions in their recommendations. A separate analysis 
was made of the 12 reviewers coming from educational and 
non-governmental agencies. The results of this analysis 
were quite similar in ratngs and recommendations to those 
from higher education institutions. 

Ormiston noted that 22 institutions received a perfect 
rating of 5 by the peer panel yet did not receive grants 
while 165 institutions with lower ratings did receive 
grants. In contrast, 11 institutions with poor or 
unacceptable ratings were funded. In terms of recommended 
funding amounts, one-third of the grants awarded were for 
less than 75 percent of the amount recommended by the peer 
panel. On the other hand, one-third received greater 
amounts than recommended by the panel. 

In drawing conclusions from the findings, Ormiston 
noted that the ratings and recommendations appeared to be 
deprived of their value because of subsequent funding 
decisions made by program officers in BIPD. He conjectured 
that legislative restrictions and other program 

considerations led to such decisions. He noted also that 
the findings support a contention that a quota exists for 
predominantly black institutions for of at least 50 percent 
of annual funds. He noted that over the eleven years of the 
program, about 54 percent of the funds had gone to black 
institutions. He also stated that there is a geographical 
factor in that for fiscal year 1975 about 56 percent of the 
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grants and 67 percent of the grant dollars went to Southern 
institutions . 

Cole, Rubin, and Cole (1977) conducted what the authors 
refer to as a sociological study of the peer review process 
in the National Science Foundation. Their study was 
conducted for the National Academy of Sciences under funding 
from NSF but with complete autonomy from that agency. The 
report reviews the NSF peer review process along with the 
types of frequent criticisms about the system from a variety 
of sources. For many critics, the main factor is the 
organizational role of the program director in funding 
decisions, the director's freedom to disregard advisory 
council recommendations, and the freedom in selecting 
reviewers . 

In order to delimit their initial efforts, the authors 
examined peer review as it operated in 10 basic research 
araas only excluding applied research and educational 
programs. Data were collected by interviewing 70 program 
directors, mail reviewers , review panel members , and related 
officials in all levels of peer review, plus reviewing the 
peer comments on 250 research proposals and related 
correspondence, and conducting a quantitative analysis of 
1,200 applicants in fiscal year 1975 when about half were 
being funded. 

Several different hypotheses were examined. One 
focused upon the charge that the "old boy" network operated 
in that eminent scientists were rated more favorable by 
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eminent reviewers than by other reviewers. Both applicants 
and reviewers were classified according to prestige of 
department from 1969 ACE ratings* The analysis showed that 
applicants from high ranked departments received slightly 
better reviews than did applicants from medium and low 
ranked departments. Using analysis of variance procedures, 
the observed mean rating for each applicant-reviewer pair 
was compared to expected mean rating assuming no bias. The 
result showing no disproportionate favoring by raters in 
high ranking departments of proposals from other high 
ranking departments. Analysis was done for each of the 10 
programs on the same issue with only one area showing more 
leniency toward high ranked departments* An analysis of 
reviewer bias in terms of geographical location of reviewer 
and relative eminence of reviewer and applicant was made. 
There was no significant tendency to favor proposals from 
one geographic area over another or for eminent scientists 
to favor proposals from other eminent scientists over less 
eminent scientists. 

A second hypothesis about the "rich getting richer" was 
examined by looking at the characteristics of the applicants 
on nine variables used to define their status in the social 
system of science. Each variable was examined separately. 
The results showed only weak or moderate correlations 
between the nine social status variables and ratings 
received on the proposals. The most highly correlated 
variable was the number of citations in the 1975 Science 
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Citation Index with only 6 percent of the ratings variance 
explained. Over all variables, only 11 percent of the 
variance was accounted for. 

The amount of agreement between mail reviewers was 
examined by looking at the mean standard deviation of 
reviewers 1 comments using the coefficient of variation. 
These ranged from .13 to .30 in the several areas. The 
results were the \ ame when correlating the mean rating as a 
dependent variable and nine independent variables. The 
authors concluded that the mail reviewers were not persuaded 
by professional status of applicants, and were more likely 
to be influenced by quality of proposed research. 

In response to the question what types of scientists 
received grants from NSF in 1975, 62 percent of those 
receiving their degrees from the highest ranked graduate 
departments received grants compared to 38 percent 

graduating from lowest- ranked departments. Further, 74 
percent of applicants currently employed in the highest 
ranked departments were funded while only 38 percent 
employed in unranked or non-academic institutions were 
funded. Recent NSF funding and citations of recent work had 
a moderate influence while professional age had almost no 
effect. 

The general structure of the findings indicated that 
scientists with an established track record, many scientific 
publications, a high frequency of citations, a record of 
having received grants from NSF plus ties to prestigious 
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academic departments result in a higher probability of 
funding than do other applicants. 

The authors introduced the sociological concept of 
-accumulated advantage" and tested it by comparing the mean 
peer re/iew ratings after dividing applicants into three 
groups; those with high, medium, and low mean ratings. 
Considering only those proposals receiving the highest peer 
ratings, estimates of probability of funding was established 
based upon the number of citations. Of the quintile with 
the highest number of citations, 100 percent received grants 
for while the lowest quintile only 77 percent received 
grants. The authors conclude here that mean peer rating was 
more important in funding than number of citations. In 
summary, the authors believe their results are consistent 
with other findings in the sociology of science that, while 
a highly stratified social system (Cole and Cole, 1973), the 
science enterprise is an equitable one favoring those who 
produce quality work* 

As Cole, Rubin, and Cole previously pointed out in 
their study, citation of published research is considered by 
many scientists to be an indicator of the value of the work 
performed. Citation in terms of the number of times a 
particular piece of research is cited as well as the total 
number of cited publications are often used as criteria for 
making judgements about the influence that a particular 
scientist has had upon a discipline. In a report on NIH 
research policies, Carter (1974) studied the validity of the 
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peer review judgements by using two measures of research 
output-approval of renewal applications and citation rates. 
The projects involved were those awarded to medical schools 
in the period 1968 to 1973. 

The first analysis made was of the relationship between 
priority scores awarded on initial application and the 
priority score on renewal applications. The correlation 
coefficient between the priority scores for the same grant 
was around +0.4. In interpreting this relationship, Carter 
suggested that the uncertain nature of research as well as 
the willingness of reviewers to be critical even of 
well-established investigators are prime factors in the low 
relationship. She noted also that the rate of disapproval 
of renewal applications declined over the period 1968-1973 
and attributed this to better quality applications. She 
also noted that the increasing approval of renewal 
applications over time provided objective evidence for 
supporting the concept of "scientific merit". In looking at 
ratings on new and earlier applications for the same 
individual, she found a statistical relationship but noted 
it was of such a nature that the major portion of the 
variance could be attributed to the merit of the project. 

The phase of the investigation involving citation data 
was done by using 747 research project grants and all 51 
program project grants awarded to medical school faculty 
competitively in fiscal 1967. information on publications 
from these grants was obtained from the £££e_ax£h — OLantS 
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Tndex and the science Citation In dex of the Institute for 
Scientific Information. The Grants .Index provided a list of 
about 5,800 publications from 1966 to 1970 while the 
Citation Index supplied a listing of all 40,000 citations 
listed in journals cited in the Citation Index from 1968-72. 
When the production of at least one frequently cited article 
was used as a citation measure, 116 grants or 15 percent of 
the total each had produced at least one of the most-cited 5 
percent of the articles in the sample. The priority scores 
on renewal applications for this set of grants was 47 point 
highe- than would have been predicted from the scores 
awarded in 1967. Carter suggested caution in using this 
finding as evidence that citations are a measure of research 
quality. She suggested that the evaluation of renewal 
applications could be strongly affected by results from the 
prior grant period. In examining the set of grants, Carter 
noted that the reviewers apparently perceived the results 
would be more useful since this set was awarded a better 
than average priority score, received larger average dollar 
awards, and had a commitment for a longer time period than 
average than did other grants in the sample. 

Recognizing that publications were from one set 
calendar period and citation rates from another set, a model 
was constructed to adjust the number of citations retreived 
to account for the year of publication. The model estimated 
the number of citations that would occur in year X after 
publication for each X in (0,6) for which data were missing. 



76 



9 

ERIC 



Page 7 3 

An estimate of the standard error of prediction of T (the 
total number of citations of an article that have or will 
occur in years 0 thru 6 following publication) was derived 
as a function of the year of publication. More than 95 
percent of T was explained by available data and the model 
for years 1966, 1967, and 1968. For articles published in 
1968, only citations for years 0-4 were available but that 
these data could predict citations in year 5 and 6 with only 
small error. 

Using average citation rates to journal articles, each 
grant was assigned to one of three categories based upon the 
principal investigator 1 s department in the medical school. 
For grants in each of the categories, the priority score 
received on the renewal was regressed on output measures 
(average citation rates, total citations, etc.). For the 
departments with lower than average citation rates, no 
output measure was found to be significantly correlated with 
the second priority score. For the basic science group and 
the medical groups, the relationships were not strong enough 
to choose one over another. Average citation rate was 
better than total citations and citation in journal articles 
appeared to be more important than citation of other 
publications. Publication count was found not to be related 
to the second priority score for any category. Carter noted 
that after citations have been included, the number of 
publications does not appear to be an additional measure of 
research quality. From the several regression analyses, the 
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variable of "average number of citations of all publications 
that were cited at least twice in the six years following 
publication" was chose to represent research quality. On 
this basis, the citation data were observed to be related to 
the priority score awarded in 1967. The author noted there 
that this relationship was further evidence that the concept 
of "scientific merit" is not completely subjective. She 
noted further that while the initial and renewal priority 
score relationship was low as noted there was a stronger 
relationship between the citation measure and the renewal 
score. 

In a subsequent paper, Carter (1978) presented data 
with regard to whether or not medical schools received money 
because of their excellence in research or because of 
favoritism. Using regression analyses with citation rate as 
a dependent variable and renewal priority score as 
predictor, the findings indicated that the average priority 
score on renewal applications was different for most 
research intensive schools after controlling for research 
output in the previous grant period. The citation data 
suggested also that the favorable judgements are explainable 
by research quality and not by being related to a research 
intensive school. With original priority score as a 
dependent variable, applications from research intensive 
schools were better even after controlling for citation 
rate. 
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The importance of the function and structure of the 
field reader and/or peer review system cannot be too highly 
stressed. The consequences of being the recipient of a 
grant or contract can be both personal and professional. 
The granting of an award can mean movement ahead in a 
research, development, training, or social program effort 
with subsequent recognition of the results. The lack of 
such funds can often mean delays in moving ahead on personal 
goals and often a diminishing of institutional rewards. 

In view of its importance, the several studies reviewed 
here have attempted to demonstrate in one form or another 
than the system does have credibility. Proposals that are 
approved and granted funds do appear to have scientific and 
technical merit at the time of funding and also produce 
useful results at some later time. Charges of cronyism, old 
boy networks and related biasing factors tend not to be 
substantiated. There is evidence that a set of prestige 
institutions and perhaps even individuals receive a large 
share of the awards but the same evidence indicates that 
these sources also are the ones producing quality research 
efforts. They have produced good work because they have 
attracted quality personnel. Thus, they have what the 
sociologists call an "accumulated advantage" in -the 
competition for funds. While there may be limitations to 
the peer review system, it appears over time to have 
developed a sufficient basis of credibility to be continued 
as the prime vehicle for reviewing and evaluating proposals 
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submitted for funding. 

VII. PROPOSAL QUALITY AND PROGRAM SUCCESS 
One aim if not the paramount aim of both informal and 
formal review processes is to aid in establishing 
relationships between the quality of a proposal and the 
resulting success of the approved program or project. The 
study by Carter on relationships between peer review 
judgements and the resulting citation of research results is 
an illustration of this objective. Two studies that sought 
to provide evidence on the relationship between proposal 
quality and subsequent program results are reviewed in this 
section. 

Proposal Quality and Final Reports 

In a study funded by the Indiana State Board of 
Vocational Education (1979), an investigation was made 
relating the quality of an initial proposal to the 
subsequent final project report. Dsing 60 projects funded 
by the SBVE during fiscal year 1976-77, both the content and 
format of the proposals and final reports were examined 
using rating scales for each dimension. A correlation of 
+.59 (p = .001) was observed between quality ratings scores 
for the proposal and final report. 

It was noted that those sections receiving the highest 
ratings in the proposal were those relating to the 
availability of specific guidelines and instructions. Those 
sections of the proposal and reports open to more 
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conceptualization tended to receive less credit. In an 
attempt to establish predictors of project quality (i.e., 
combined proposal and final report ratings) , the proposals 
were categorized by the presence or absence of credit for 
several items representing sections of the proposal- Overall 
mean scores were then compared for items receiving credit 
and those not receiving credit. Only the item "Objectives 
are Clearly Written and Specific" was found to be 
statistically significant using a two-tailed t-test. It is 
interesting to note that in the table reporting these 
results, the item "Procedures are Provided with Sufficient 
Detail" had a larger between-means difference yet no results 
are presented with regard to the outcome of the statistical 
test for this item. 

In addition to the above, an investigation was made of 
the readability of both proposal and report formats and 
their comprehensiveness. It was found that the mean scores 
for readability were higher than for comprehensiveness. 
Because of the frequency with which they occurred in the 
proposals, an examination was also made regarding the role 
of advisory committees, literature reviews, and 

instrumentation. The results showed that, in general, 
insufficient information about these items was present in 
both proposals and final reports. 

In the summary, the report stressed a need for more 
specific guidelines in the areas relating to the 

conceptualization of the research studies. They noted also 
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tftat their analyses demonstrated that various sections of a 
proposal can give indications of the subsequent quality of a 
proposed project. 

Proposal Quality and Program Implementation 

Recognizing the importance of hav ing successful 
pro j ect s as a means of accompl ishing program objectives, 
Toia (1974) investigated the relationship between four 
factors which might affect both securing an award and the 
successful conduct of program implementation. The four 
factors were the administrative relationship existing 
between the grantee agency and the local government, the 
educational and prior experience of the professional staff 
of the grantee agency, the amount and type of technical 
assistance used in preparing the proposal and in program 
implementation, and the similarity of staff characteristics 
to client characteristics. These four factors or criteria 
were related to the quality of the proposal and the success 
of the program implementation after funding. A group of 16 
proposals funded in Fiscal Year 1971 under the Youth 
Development and Delinquency Prevention Administration 
constituted the data for analysis. Each proposal was ranked 
by five panel members independently and then in a joint 
session. A quality rating score and final rank for each 
proposal was obtained by summing the rankings for the five 
panelists. A measure of the quality of program 

implementation was developed and submitted to the project 
directors &nd non-clerical employees of the project. Raw 
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scores for each ^gency on the independent variables was 
establisl^ and then correlated by multiple regression 
separately for the independent variables. In addition, the 
correlations were obtained for quality rating and rank on 
proposal and program implementation. 

The analysis revealed that the correlations between 
proposal quality and program implementation ranks was 
negative (-.33797) using an interval scale approach and 
-.3367 using a rank order analysis. The author attributes 
the negative relationship to the possible use of consultant 
proposal writers possessing little or no relation to the 
real world and who may have focused on developing a proposal 
that would "sell". 

The relationship between program implementation ratings 
and the four independent variables showed all four as 
significant with educational background of staff accounting 
for 46 percent of the variance. For all four factors, 75 
percent of the variance was accounted for with 

administrative relationships the second variable followed by 
technical assistance and then personal characteristics. 
These same four variables ■ hen correlated with proposal 
quality showed no significant relationships with technical 
assistance only accounting for most of the variance (4 
percent) . Adding the other variables accounted for only 8 
percent of the total variance. 

In discussing the findings, the author noted the 
discrepancy between proposal quality and implementation 
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ratings and indicated that the proposal quality rating was a 
poor and imperfect predictor of program implementation 
success. The author stated that those agencies who invested 
in their professional staffs appeared to be the ones most 
1 ikely to be successful . 
Commentary 

It is interesting to observe the lack of studies 
relating proposal quality to subsequent program success. In 
view of the interest in increasing predictability, the 
results presented are not consistent, in one case, the 
relationship between proposal and final report was positive 
while in the other the relationship between proposal and 
program success was negative. The positive result might be 
explained by the similarity between proposal and final 
report components. If the objectives are well stated in the 
former they are likely to be also in the latter. In 
contrast, the factors examined in re] ationship to proposal 
development (such as Technical Assistance) could very well 
not be related to the kinds of efforts needed to make a 
program successful. Thus similarity of variables examined 
leads to a positive result while dissimilarity leads to 
negative results. 
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VIII. PERCEPTIONS AND ATTITUDES ABODT PROPOSAL DEVELOPMENT 
Proposal development and evaluation, like many another 
process generates a series of beliefs, attitudes, and 
perceptions about what it takes to be successful. One often 
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hears stories about how an individual received a large 
amount of funds by simply sending in a proposal on the back 
of a post card. There are also stories that RFPs merely 
comply with bid competition and that the contract for 
substance of the RFP has been "wired" - some agency or 
individual has previously been identified as the winner. At 
the same time, there are some realities to proposal 
development. Certainly one is the missing of a mail or 
submission deadline resulting in a rejection of the 
proposal. Another would focus around the failure to 
properly read a program ammouncement and thus not respond to 
a priority area. Two studies were identified that 
investigated this area of proposal development. One study 
focused upon a general recommendation made to proposal 
developers while the second focused securing attitudes 
toward the overall process of development and evaluation. 

One common recommendation in the literature is that 
prospective proposal developers take time to review what an 
agency has funded in the past as a guide to knowing if one 
should submit their ideas to that agency. 

One study relevant to this point was done by Siegel 
(1977) in securing perceptions held by agencies, who were 
often the recipients of funds from foundations, as to the 
factors which governed ace— tance or rejection of proposals 
by such foundations. Using a questionnaire approach, 90 
agencies in Franklin County, Ohio were solicited with regard 
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to reasons perceived by them for acceptance or rejection of 
proposals by foundations. Seven research questions directed 
the s.-udy. Sixty-eight agencies returned the questionnaire 
for a return rate of 76 percent. One follow-up was made. 
Thirty-nine of the completed forms were from private 
agencies seeking funds, 24 from public, and 5 from 
quasi-public agencies. Demographic data regarding 

responding agencies are presented in the report. Of the 
group responding, 48 or about 72 peLCent had applied for a 
grant but 19 had not. Of the 48 applications, 28 had 
received and 20 had not received a grant. Data is based 
only upon the 48 responding Yes and consists of descriptive 
statistics and Chi-square. 

With regard to source of funds, 32 or 71 percent felt 
that their chancer were best to get money from the local 
level as opposed to other levels. As to topics most easily 
funded, the general finding was that grants for the 
handicapped were easiest followed by child abuse. With 
regard to type of grant (on-going, one-time, or matching), 
on-going grants were perceived as being the most difficult 
to secure (37 percent) and one-time grants the easiest. 
Seventy-five percent felt that proposal writing was 
necessary administrative skill within an agency seeking 
funds. As for importance of the various sections of the 
proposal, the specification of objectives and the budget 
were perceived as being most important (77 and 79 percen- 
respectively) to funding agencies. 
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In terms of perceived reasons for rejection, the 
highest percentage was for the request being improper or 
ineligible (36 percent), lack of planning for future 
spending (38 percent), lack of measurable need (33 percent), 
staff experience (26 percent), with other reasons receiving 

fewer responses. 

Respondents were also asked to rate a series of items 
expressing views about proposal preparation as they relate 
to securing foundation grants. Of the 48 respondents, 42 
Agreed or Strongly Agreed with the statement "Knowing 
foundation staff contributes to grant acceptance"; 33 
percent Agreed or Strongly Agreed with the statement 
"Getting foundation proposals accepted usually involves 
political considerations"; 36 were either Uncertain or 
Agreeing with the statement "Who you are, as an agency, 
determines grant acceptance"; 35 responded similarly to the 
statement "There is a formula for getting proposals 
accepted"; 29 responded Uncertain or Disagreeing with the 
statement "There is a diverse community representation on 
most foundations boards"; and the responses were about 
equally divided between Disagreeing and Agreeing with the 
statement that "There is a mystification surrounding grant 
proposals" . 

In terms of the original seven reQearch questions, 
Siegel makes the following summary: On-going grants were 
perceived as being the most difficult to secure; most 
agencies that do not actively research foundations do not 
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get proposals accepted; grant proposals are rejected most 
frequently due to improper or ineligible requests; getting 
foundations proposals accepted usual ly involves pol itical 
considerations; knowing foundation staff personnel 

contributes to grant acceptance; there is a formula for 
getting foundation proposals accepted; and that the 
introduction section of a proposal was not most important to 
the funding agency. 

MYthfi flPrf Realities 

Recognizing that proposal development may have its 
myths and realities, Cook and Loadjnan (1982) initiated 
development work on instrumentation to assess perceptions 
and attitudes about proposal development and evaluation. 
Drawing upon personal experiences and the large literature 
base on proposal development, a series of statements were 
created to reflect both myths and realities. An initial set 
of 86 statements was created and administered to individuals 
at the university level attending workshops and enrolled in 
courses on proposal development. Using facto r analysis 
procedures, the statements were reduced to a final set of 54 
items scaled in Likert format with a score of 1 representing 
Strong Agreement and 5 representing Strong Disageement with 
the statement. The final set of scaled items was mail 
administered to a systematic sample of 419 individuals 
listed in the 1979 Biographical Membership Directory of the 
American Educational Research Association. A total of 231 
subjects returned usable responses. Each respondent was 
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asked to provide data with regard to proposal development 
experience, membership on peer panels, operation of 
projects, and the conduct of proposal training sessions. 
The responses of the 231 subjects were factor analyzed and 
five factor scores generated. The reliability of the five 
factors ranged from +.49 to +.83. Respondents were 
classified into groups based upon their proposal development 
and peer panel experience. Discriminant analyses were made 
but the resulting classification functions did not predict 
group membership at anything better than a chance level. 
Consequently, emphasis was given to examination of the items 
as contrasted to factor scores. 

Using the 54 items as predictor variables and 
classification variables of peer panel experience (including 
proposal development) against no peer panel experience, a 
stepwise discriminant function was made. Of the total item 
set, 19 items correctly classified group membership at a 72 
percent level. The resulting analysis suggested that the 
perceptions of persons having had peer panel experience 
differed in their responses to the instrument than those who 
had not had the experience. To develop some sense cf myth 
and reality, items were classified using the mean score from 
all respondents into endorsed (high agreement) , non-endorsed 
(lorf agreement), or neutral statements. There were nine 
items receiving strong endorsement. They were viewed as 
representing reality and are as follows in shortened form: 
- know the funding source 
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- write clearly and precisely 

- the proposing agency reputation makes a 
difference 

- the understandability of the proposal 
is important 

- staff capability is important: 

- documentation of costs is essential in 
budget preparation 

- developing a proposal does not guarantee 
funding 

- there should be flexibility in developirg 
the workscope 

- you cannot miss the deadline for submitting 
a proposal 

There were seven items receiving low endorsement 
represented by their mean score. They are viewed 
representing mythology and are as follows in shortened fo 

- there is a stigraa associated with not 
being funded 

- the grant process is intentionally 
difficult 

- small agencies 1 probability of obtaining 
continued grant support is low 

- who you know is more important than the 
the quality of the proposal 

- proposal content should be purposely left 
vague 
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- proposal development should be done by a 
single indi v idual 

- professional grant writers would be 
employed to write proposal s 

Based upon the results of the analysis, the authors 
concluded that it was possible to develop instrumentation 
that would function reasonably well in assessing perceptions 
about proposal development. There did appear to be some 
statements that are endorsed as reality and some endorsed as 
myths. In addition, there are differences in responses 
between those with peer panel experience and those who have 
not had such an experience. 

It should be noted that the instrument development was 
carried out prior to the investigation reported in this 
document. Many of the realities and myths as detected in 
the earlier study have received support from the findings of 
the empirical studies cited in this report. A next step is 
to combine the results of the two investigations and explore 
further the process of proposal development in order to more 
firmly establish principles supporting the process. 
CsumSJliJLXy. 

The two studies reviewed here suggest that individuals 
involved in the process of proposal development are able to 
make reasonable perceptions and -o hold valid attitudes 
towards what works and doesn't work in the process. 
Although working vitbin a limited population, Sicgel was 
able to note that potential proposal initiators have an idea 
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of what the agencies involved would fund and not fund. As 
for attitudes which develop about the process, the viability 
of developing instrumentation which would assist in sorting 
out myths and realities regarding the process seems 
reasonable. Both studies suggest that potential fund 
seekers have a sense of reality about their pursuit of such 
funds . 

IX. A SYNTHESIS 
In a recent article on the variety of mathematical 
models, Karplus (1983) identified three types of problems 
with which systems engineers and scientists deal. He does 
this by using the concepts of excitation , response, and 
s ystem . Problems of anal ysis are those where the excitation 
and the system are given and the task is to find the 
response. In the case of synthesis , the excitation and 
response are given and the system involving the relationship 
is to be found or realized. In the third type of problem, 
the system and the response are given and the task is to 
find the excitation. The latter type are considered as 
instrumentation or £QJDl£X&1 problems. 

In developing a synthesis for this pape r , the 
relationships noted above will serve as a metaphor. In the 
proposal case , there is an excitatior in that there are 
conditions which stimulate or excite individuals to develop 
proposals, (e.g., program announcements, RFPs) . There are 
responses in that some proposals become operating projects. 
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The prime interest here was the system between the 
excitation and the response with the aim to secure a better 
understanding of the H black box" of proposal development 
based upon empirical research. This section focuses upon 
drawing some salient observations about the proposal 
development and evaluation process by synthesizing findings 
from the set of studies reviewed in this paper* Statements 
relative to both methodology of investigation as well as 
substantive findings are presented with the latte. being 
presented first. 
Observations on Substanc e 

In setting forth the synthesis of substantive 
observations, statements are presented which are integrative 
in that they may draw from one or more studies. With this 
condition as background, the following observations appear 
to have some empirical basis: 

- Even though the major part of proposal develop- 
ment, there are very few empirical studies 
directed toward the task of actual proposal 
preparation. Proposal developers draw upon 
their own experience to develop the creative 
and conceptual elements of a proposal. 

- Support se rvices provided to proposal 
developers take a variety of forms but those 
found to be most useful focus upon assistance 
in developing the somewhat mechanical aspects 
of a proposa] , such as budgets, duplication, 
and similar items . 

- The task of proposal preparation can be a 
contributing factor or influence on changing 
organizational behav io ral patterns. 

- The source of proposal development support 
tends to be in an area immediate to the 
proposal developer such as the department 
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of assignment* 

- The general distribution to interested 
parties of fund availability as well as 
information targeted to specific persons 
appears to be a justifiable institutional 
procedure. 

- Training in proposal development is an 
activity deemed a justifiable service and 
cost. 

- The costs of proposal development vary 
according to a set of variables such as the 
type of proposal , the agency from which funds 
are sought, the product to be produced, and 
the size of the proposal. 

- Development cost estimates based upon experi- 
ence tend to be positively related to actual 
costs derived by empirical procedures. 

- The return on the investment as derived from 
funded proposals although somewhat low 
percentage-wise nevertheless justifies the 
costs of development. 

- The decision points and responsibilities in 
the process of proposal development within an 
institution should be the object of careful 
study and clearly identified. 

- The phase of proposal development receiving 
the greatest attention has been the review 
and evaluation process, especially the peer 
review system. 

- Funding agencies develop both informal and 
formal procedures for screening applications 
to be reviewed. 

- The quality of proposals in terms of scientific 
and technical merit appears to be the most 
important consideration in the peer review 
process. 

- Based upon information presented in the panel 
sessions, raters have been found to change 
their ratings with the content of the informa- 
tion presented being more important than the 
expertness of the individual presenting it. 

- Procedures can be developed which can increase 
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the inter-rater reliability of peer ratings. 

- The concept of "scientific merit" as a factor 
upon which funding decisions are made appears 
to be a valid one for making such decisions. 

- There appear to be instances where political- 
social considerations tend to override the 
worth of proposals even as judged by peers. 

- There appears to be no consistent pattern of 
factors or variables which distinguish between 
proposals that are accepted or rejected when 
compa r isons are es tabl ished. 

- Information feedback to rejected applicants 
varies from specific, usable comments to no 
information of value. 

- Charges of favoritism, cronyism, old boy 
network as influencing factors in award 
decisions are not supported to any strong 
degree. 

- Applicants from high ranked or prestigious 
departments have an "accumulated advantage" 
in that the research issuing from such 
departments is generally of higher quality 
and this is reflected in the proposals sub- 
mitted. 

- Citation of work produced under a proposal 
tends to be positively related with initial 
and renewal ratings. 

- Eminence as a scientist appears not in it- 
self to guarantee funding. 

- An individual's position in the social 
strata of science was found to have a 
positive but low relationship to ratings 
received on pr oposal s. 

- The peer review system while having some 
limitations appears to be substantiated as 
a viable means for establishing the 
scientific and technical merit of research 
proposal s. 

- Relationships between proposal quality 
ratings and project implementation tend to 
be inconsistent. 
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- Developing familiarity with the agency from 
which funds are sought is a valid behavior 
since there are often misperceptions by 
clients as to what is important to the 
agency . 

- Individuals experienced in proposal develop- 
ment, including peer panel experience, tend 
to view the system more favorably than those 
who have not submitted applications or who 
expe rienced re j ection, 

- Instrumentation can be developed useful in 
assessing the realities and mythology 

sur rounding proposal development and 
evaluation. 

Observations on Methodology 



Studies and reports included in this study were 



principally those in which the investigator stated a 
question or hypothesis and then developed a procedure or 
method to collect quantitative data to answer the question 
or hypothesis. As a consequence, many studies of what some 
would call qualitative or naturalistic inquiry are not 
included. Using the studies and reports actually cited, 
several observations can be made upon the methodological 
dimensions employed. 

- Research of an experimental or variable 
manipulation foriri was not a major form 
of method. 

- The predominant me the a of analysis tends 
to be some form of correlational analysis 
involving techniques such as regression 
analysis , discriminant analysis , one-way 
and multivariate analyses of variance, and 
path analysis . 

- Survey methods were employed in the form of 
personal interviews, completion of self- 
report forms, mail surveys, telephone calls, 
or public hearings. 
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- Historical and archival methods were used to 
develop background material for surveys as 
well as providing basic data. 

- In many analyses, the dependent variable was 
often the rating or score assigned to a 
proposal with other variables such as 
presence or absence of a proposal component, 
professional status, publ ication rates , 
serving as independent variables. 

X. CONCLUSIONS AND IMPLICATIONS 

The prime objective of this study was to examine the 

existing literature relating to proposal development and 

evaluation in order to establish a perspective on any 

empirical base underlying the process. Findings relative to 

various aspects of the overall process were presented in 

previous sections of this report. Based upon those 

findings, a synthesis of observations regarding both the 

substantive nature of the studies as well as their 

methodological approaches was presented. Dsing the 

synthesis as a starting point, several conclusions may be 

drawn. 

The empirical research base supporting the task of 
proposal development find evaluation is uneven. There are 
few studies supporting actions taken with regard to the 
process of proposal development and preparation. In 
contrast, there is a fairly large number of studies relating 
to the evaluation process, particularly with regard to the 
use of peer panels and the validity of their judgements. 
Thus one can feel more secure about statements made relative 
to proposal evaluation than one can with regard to proposal 
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development • 

The research reported both in the form of studies and 
reports focuses primarily upon those activities subject to 
enumeration, such as the frequency of support services 
utilized, preparation costs, and similar aspects. This 
conclusion is to some degree a condition of the literature 
reviewed since only studies of that type were reviewed. 
Nevertheless, the point to be made here is that the 
dimensions investigated are those that can be subjected to 
quantitative treatment of data. To support this conclusion, 
it should be noted that a comprehensive search of the 
literature uncovered only one or two studies that might 
qualify as qualitative investigations. 

While there is some evidence that factors other than 
scientific merit sometimes enter into the evaluation of 
proposals, the general conclusion can be drawn that the 
system is trustworthy and does result in a high level of 
quality proposals. The proposal developer can by and large 
have faith that a fair review of the submitted proposal war 
made. The findings also imply that if an individual wanted 
to be a more consistent winner then affiliation with a group 
of colleagues of sufficient high caliber would result in 
significant ideas worthy of funding. 

To summarize, even though a set of studies was 
identified relating to the task of proposal development and 
evaluation, one is left with the feeling that the movement 
from an idea to the documentation of that idea in the form 
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of a proposal is essentially a creative act and therefore 
not highly amenable to empirical investigation. Until such 
time as creative acts can be subjected to empirical 
methodologies, that aspect of proposal development will have 
to more or less operate from a rather personal, intuitive 
basis rather than upon an empirical knowledge base. Thus, 
the current s tate-of -the-lcnowledge is rather limited in both 
its scope and established principles- 

The principal disciplines that have initiated studies 
relating to the proposal development and evaluation have 
been those involving the natural and physical sciences. 
There has been much less study done in the social sciences 
area with even less done in the field of education. Because 
cf the importance of individuals receiving support for 
continuing research programs and efforts and its effect on 
subsequent professional status, the research on proposals in 
the sciences has tended to be limited to the peer review 
process and award decisions. 

Even though the peer review process is the continuing 
source of controversy (Anderson, 1983)/ the utilization of 
peers to judge the technical and/or scientific merit of 
proposals has validity. Charges of favoritism or similar 
biasing factors tend not be substantiated. The perception 
that the same individuals and institutions are continuous 
winners is based more upon the accumulated advantage 
accruing to the institution in the degree that it attracts 
quality personnel who develop high quality proposals- 
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Given the conclusions drawn above, what are some 
implications for the practice of proposal development and 
evaluation? It would appear that one implication is that 
proposal preparation will still have to rely more upon the 
"art" side of the task than upon the "science" side in view 
of the limited empirical evidence to support actions 
undertaken. A second implication relates to the continued 
provision of support service to those persons developing 
proposals. Since many proposal writers view these support 
services primarily for their mechanical cont ributions , 
perhaps efforts need to be made to see how such services can 
make contributions to the more creative aspects of proposal 
preparation. Evidence from the peer review findings suggest 
that if an institution would like to become a winner in the 
game of proposal funding, then efforts should be directed 
toward building prestigious departments wherein innovative 
ideas can be developed between and among individuals. Such 
an action would aid in building a foundation for a "track 
record" of quality proposal development. Regardless of the 
path chosen, the findings of this investigation support the 
investment of resources to acquire new funds since the 
return on such investment, while sometimes low, is 
nevertheless in a positive direction. 
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APPENDIX A 
Profile of Searc h Terms 



Grant Application 
Grant Development 
Grant Review 
Peer Panel 
Peer Review 
Proposal Development 
Proposal Writing 
Proposal Evaluation 
Proposal Preparation 
Proposal Review 
Grantsmanship 
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